Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251023 20:15:10.831254 23038 test_util.cc:276] Using random seed: 386858599
2025-10-23T20:15:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-23T20:15:10Z Disabled control of system clock
I20251023 20:15:10.864701 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.190:34331
--webserver_interface=127.22.127.190
--webserver_port=0
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251023 20:15:11.185736 23046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:11.186034 23046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:11.186108 23046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:11.194217 23046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:11.194312 23046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:11.194365 23046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:11.194411 23046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:11.204380 23046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.190:34331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.127.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
I20251023 20:15:11.206007 23046 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:11.207751 23046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:11.215816 23052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:11.215840 23051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:11.216578 23054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:11.217034 23046 server_base.cc:1047] running on GCE node
I20251023 20:15:11.217796 23046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:11.219200 23046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:11.220396 23046 hybrid_clock.cc:648] HybridClock initialized: now 1761250511220345 us; error 39 us; skew 500 ppm
I20251023 20:15:11.220764 23046 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 20:15:11.222908 23046 webserver.cc:492] Webserver started at http://127.22.127.190:44199/ using document root <none> and password file <none>
I20251023 20:15:11.223635 23046 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:11.223740 23046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:11.224077 23046 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:11.226562 23046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/instance:
uuid: "6545140d94b7485aa81c51ad233355cd"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
I20251023 20:15:11.227334 23046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal/instance:
uuid: "6545140d94b7485aa81c51ad233355cd"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
I20251023 20:15:11.232587 23046 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251023 20:15:11.236073 23060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:11.237434 23046 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251023 20:15:11.237596 23046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
uuid: "6545140d94b7485aa81c51ad233355cd"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
I20251023 20:15:11.237784 23046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:11.253768 23046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:11.254647 23046 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:11.254909 23046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:11.272934 23046 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.190:34331
I20251023 20:15:11.272965 23111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.190:34331 every 8 connection(s)
I20251023 20:15:11.274438 23046 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:11.278678 23112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:11.281733 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 23046
I20251023 20:15:11.281983 23038 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal/instance
I20251023 20:15:11.284416 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.189:33509
--webserver_interface=127.22.127.189
--webserver_port=0
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251023 20:15:11.284505 23112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:11.301532 23112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } has no permanent_uuid. Determining permanent_uuid...
W20251023 20:15:11.302704 23061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111)
W20251023 20:15:11.304201 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.189:33509: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111)
I20251023 20:15:11.353508 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } attempt: 1
W20251023 20:15:11.355460 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.189:33509: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111)
I20251023 20:15:11.423703 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } attempt: 2
W20251023 20:15:11.425557 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.189:33509: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111)
I20251023 20:15:11.516799 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } attempt: 3
W20251023 20:15:11.518520 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.189:33509: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111)
W20251023 20:15:11.619094 23114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:11.619392 23114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:11.619504 23114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:11.628093 23114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:11.628196 23114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:11.628309 23114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:11.628376 23114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:11.638859 23114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.189:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.127.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
I20251023 20:15:11.640476 23114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:11.642333 23114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:11.650242 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:11.650271 23122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:11.650871 23125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:11.651316 23114 server_base.cc:1047] running on GCE node
I20251023 20:15:11.652058 23114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:11.653604 23114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:11.654807 23114 hybrid_clock.cc:648] HybridClock initialized: now 1761250511654759 us; error 44 us; skew 500 ppm
I20251023 20:15:11.655159 23114 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 20:15:11.657334 23114 webserver.cc:492] Webserver started at http://127.22.127.189:42879/ using document root <none> and password file <none>
I20251023 20:15:11.658075 23114 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:11.658177 23114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:11.658509 23114 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:11.661448 23114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/instance:
uuid: "ae5a310b4a514c0f812bda133c756062"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
I20251023 20:15:11.661727 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } attempt: 4
I20251023 20:15:11.662293 23114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal/instance:
uuid: "ae5a310b4a514c0f812bda133c756062"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
W20251023 20:15:11.663492 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.189:33509: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111)
I20251023 20:15:11.667847 23114 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20251023 20:15:11.671257 23131 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:11.672690 23114 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:11.672878 23114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
uuid: "ae5a310b4a514c0f812bda133c756062"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
I20251023 20:15:11.673120 23114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:11.687219 23114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:11.688130 23114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:11.688387 23114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:11.709877 23114 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.189:33509
I20251023 20:15:11.709908 23182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.189:33509 every 8 connection(s)
I20251023 20:15:11.711352 23114 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/info.pb
I20251023 20:15:11.711740 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 23114
I20251023 20:15:11.711921 23038 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal/instance
I20251023 20:15:11.713245 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.188:41417
--webserver_interface=127.22.127.188
--webserver_port=0
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251023 20:15:11.715929 23183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:11.724241 23183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:11.738734 23183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:11.748063 23183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
W20251023 20:15:11.749173 23132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
W20251023 20:15:11.750801 23183 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:11.800109 23183 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 1
W20251023 20:15:11.802066 23183 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:11.870296 23183 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 2
W20251023 20:15:11.872125 23183 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:11.962796 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } attempt: 5
I20251023 20:15:11.963337 23183 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 3
W20251023 20:15:11.965335 23183 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:11.970119 23112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
W20251023 20:15:11.971920 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:12.023182 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 1
W20251023 20:15:12.025249 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
W20251023 20:15:12.072938 23185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:12.073277 23185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:12.073356 23185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:12.082139 23185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:12.082244 23185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:12.082300 23185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:12.082346 23185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:12.092736 23185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.188:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.127.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
I20251023 20:15:12.093504 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 2
I20251023 20:15:12.094504 23185 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20251023 20:15:12.095331 23112 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:12.096482 23185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:12.104312 23194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:12.104290 23193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:12.105471 23196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:12.105814 23185 server_base.cc:1047] running on GCE node
I20251023 20:15:12.106516 23185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:12.107935 23185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:12.108575 23183 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 4
I20251023 20:15:12.109211 23185 hybrid_clock.cc:648] HybridClock initialized: now 1761250512109114 us; error 31 us; skew 500 ppm
I20251023 20:15:12.109617 23185 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20251023 20:15:12.110337 23183 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.127.188:41417: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:12.111877 23185 webserver.cc:492] Webserver started at http://127.22.127.188:41587/ using document root <none> and password file <none>
I20251023 20:15:12.112602 23185 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:12.112687 23185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:12.112986 23185 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:12.119755 23185 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data/instance:
uuid: "690ffde75cd745b6a87925967d54471a"
format_stamp: "Formatted at 2025-10-23 20:15:12 on dist-test-slave-g5bj"
I20251023 20:15:12.120559 23185 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal/instance:
uuid: "690ffde75cd745b6a87925967d54471a"
format_stamp: "Formatted at 2025-10-23 20:15:12 on dist-test-slave-g5bj"
I20251023 20:15:12.126226 23185 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251023 20:15:12.129813 23202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:12.131333 23185 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251023 20:15:12.131510 23185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal
uuid: "690ffde75cd745b6a87925967d54471a"
format_stamp: "Formatted at 2025-10-23 20:15:12 on dist-test-slave-g5bj"
I20251023 20:15:12.131714 23185 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:12.151489 23185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:12.152462 23185 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:12.152738 23185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:12.175040 23185 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.188:41417
I20251023 20:15:12.175074 23253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.188:41417 every 8 connection(s)
I20251023 20:15:12.176663 23185 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data/info.pb
I20251023 20:15:12.181252 23254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:12.181898 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 23185
I20251023 20:15:12.182098 23038 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal/instance
I20251023 20:15:12.189561 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:12.201539 23112 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 3
I20251023 20:15:12.203784 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:12.215174 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } has no permanent_uuid. Determining permanent_uuid...
I20251023 20:15:12.218030 23112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Bootstrap starting.
I20251023 20:15:12.222021 23112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:12.224027 23112 log.cc:826] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:12.226549 23112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: No bootstrap required, opened a new log
I20251023 20:15:12.226608 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Bootstrap starting.
I20251023 20:15:12.229691 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:12.231030 23254 log.cc:826] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:12.232376 23112 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.232657 23112 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:12.232844 23112 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6545140d94b7485aa81c51ad233355cd, State: Initialized, Role: FOLLOWER
I20251023 20:15:12.233505 23112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.233700 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: No bootstrap required, opened a new log
I20251023 20:15:12.234922 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.235203 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:12.235890 23112 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:12.240249 23254 raft_consensus.cc:359] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.240563 23254 raft_consensus.cc:385] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:12.240675 23254 raft_consensus.cc:740] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690ffde75cd745b6a87925967d54471a, State: Initialized, Role: FOLLOWER
I20251023 20:15:12.241335 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.242223 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.242588 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:12.243731 23254 sys_catalog.cc:565] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: configured and running, proceeding with master startup.
W20251023 20:15:12.247772 23276 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:15:12.248112 23276 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251023 20:15:12.248235 23046 master_runner.cc:186] Error getting master registration for 127.22.127.189:33509: OK, instance_id { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" instance_seqno: 1761250511704082 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251023 20:15:12.257040 23046 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251023 20:15:12.257483 23066 tablet.cc:2378] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:12.264782 23185 master_runner.cc:186] Error getting master registration for 127.22.127.189:33509: OK, instance_id { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" instance_seqno: 1761250511704082 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251023 20:15:12.269425 23185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251023 20:15:12.275630 23288 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251023 20:15:12.275794 23288 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251023 20:15:12.404992 23208 tablet.cc:2378] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:12.409535 23183 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } attempt: 5
I20251023 20:15:12.423784 23183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Bootstrap starting.
I20251023 20:15:12.428090 23183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:12.429673 23183 log.cc:826] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:12.433543 23183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: No bootstrap required, opened a new log
I20251023 20:15:12.443095 23183 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.443485 23183 raft_consensus.cc:385] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:12.443612 23183 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5a310b4a514c0f812bda133c756062, State: Initialized, Role: FOLLOWER
I20251023 20:15:12.444402 23183 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.446084 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.446460 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:12.447433 23183 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:12.453439 23291 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:12.453709 23291 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.455482 23291 leader_election.cc:290] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:12.456605 23229 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a" is_pre_election: true
I20251023 20:15:12.457212 23229 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5a310b4a514c0f812bda133c756062 in term 0.
I20251023 20:15:12.457868 23132 leader_election.cc:304] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 690ffde75cd745b6a87925967d54471a, ae5a310b4a514c0f812bda133c756062; no voters:
I20251023 20:15:12.458333 23291 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:12.458467 23291 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:12.458560 23291 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:12.459272 23087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6545140d94b7485aa81c51ad233355cd" is_pre_election: true
I20251023 20:15:12.459748 23087 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5a310b4a514c0f812bda133c756062 in term 0.
I20251023 20:15:12.461810 23291 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.462914 23087 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6545140d94b7485aa81c51ad233355cd"
I20251023 20:15:12.462982 23229 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a"
I20251023 20:15:12.463126 23087 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:12.463184 23229 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:12.466598 23229 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5a310b4a514c0f812bda133c756062 in term 1.
I20251023 20:15:12.466596 23087 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5a310b4a514c0f812bda133c756062 in term 1.
I20251023 20:15:12.467214 23132 leader_election.cc:304] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd, 690ffde75cd745b6a87925967d54471a, ae5a310b4a514c0f812bda133c756062; no voters:
I20251023 20:15:12.469597 23291 leader_election.cc:290] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 1 election: Requested vote from peers 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:12.469880 23291 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:12.470031 23291 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 1 LEADER]: Becoming Leader. State: Replica: ae5a310b4a514c0f812bda133c756062, State: Running, Role: LEADER
I20251023 20:15:12.470712 23291 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:12.477679 23302 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251023 20:15:12.480593 23303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5a310b4a514c0f812bda133c756062. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.480919 23303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:12.492691 23305 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:12.498095 23305 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:12.514168 23229 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 1 FOLLOWER]: Refusing update from remote peer ae5a310b4a514c0f812bda133c756062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:12.515295 23303 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:12.519928 23087 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Refusing update from remote peer ae5a310b4a514c0f812bda133c756062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:12.520942 23303 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:12.544485 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5a310b4a514c0f812bda133c756062. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.544817 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:12.551247 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5a310b4a514c0f812bda133c756062. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.551501 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:12.555035 23307 mvcc.cc:204] Tried to move back new op lower bound from 7214082099244572672 to 7214082099085787136. Current Snapshot: MvccSnapshot[applied={T|T < 7214082099244572672}]
I20251023 20:15:12.563225 23303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.563474 23303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:12.563684 23303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.563840 23303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:12.565670 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.565899 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:12.565917 23308 mvcc.cc:204] Tried to move back new op lower bound from 7214082099244572672 to 7214082099085787136. Current Snapshot: MvccSnapshot[applied={T|T < 7214082099244572672}]
I20251023 20:15:12.566300 23305 catalog_manager.cc:1357] Generated new cluster ID: 67ec0cebd86e4322972b698542d64fab
I20251023 20:15:12.566404 23305 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:12.577373 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:12.577607 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:12.598719 23305 catalog_manager.cc:1380] Generated new certificate authority record
I20251023 20:15:12.600667 23305 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:12.621265 23305 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Generated new TSK 0
I20251023 20:15:12.622457 23305 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:15:12.639990 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.127.129:0
--local_ip_for_outbound_sockets=127.22.127.129
--webserver_interface=127.22.127.129
--webserver_port=0
--tserver_master_addrs=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251023 20:15:13.119410 23317 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae5a310b4a514c0f812bda133c756062)
I20251023 20:15:13.119650 23317 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:13.121210 23317 leader_election.cc:290] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:13.121803 23228 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a" is_pre_election: true
I20251023 20:15:13.121927 23158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae5a310b4a514c0f812bda133c756062" is_pre_election: true
I20251023 20:15:13.122727 23061 leader_election.cc:304] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd; no voters: 690ffde75cd745b6a87925967d54471a, ae5a310b4a514c0f812bda133c756062
I20251023 20:15:13.123255 23317 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:13.164947 23316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:13.165390 23316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:13.165580 23316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:13.178488 23316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:13.178732 23316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.127.129
I20251023 20:15:13.193769 23316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.127.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.127.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.127.129
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
I20251023 20:15:13.195549 23316 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:13.197813 23316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:13.209465 23323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:13.209723 23325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:13.209551 23322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:13.211470 23316 server_base.cc:1047] running on GCE node
I20251023 20:15:13.212011 23316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:13.212939 23316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:13.214134 23316 hybrid_clock.cc:648] HybridClock initialized: now 1761250513214080 us; error 45 us; skew 500 ppm
I20251023 20:15:13.214430 23316 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 20:15:13.216691 23316 webserver.cc:492] Webserver started at http://127.22.127.129:34533/ using document root <none> and password file <none>
I20251023 20:15:13.217444 23316 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:13.217545 23316 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:13.217953 23316 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:13.220221 23316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data/instance:
uuid: "d220aba7c9c0400398ffde917376ee48"
format_stamp: "Formatted at 2025-10-23 20:15:13 on dist-test-slave-g5bj"
I20251023 20:15:13.220973 23316 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/instance:
uuid: "d220aba7c9c0400398ffde917376ee48"
format_stamp: "Formatted at 2025-10-23 20:15:13 on dist-test-slave-g5bj"
I20251023 20:15:13.227155 23316 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20251023 20:15:13.230836 23331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:13.232154 23316 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:13.232311 23316 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal
uuid: "d220aba7c9c0400398ffde917376ee48"
format_stamp: "Formatted at 2025-10-23 20:15:13 on dist-test-slave-g5bj"
I20251023 20:15:13.232508 23316 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:13.247301 23316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:13.248167 23316 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:13.248414 23316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:13.249544 23316 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:13.250890 23276 catalog_manager.cc:1269] Loaded cluster ID: 67ec0cebd86e4322972b698542d64fab
I20251023 20:15:13.251050 23276 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: loading cluster ID for follower catalog manager: success
I20251023 20:15:13.251582 23316 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:13.251675 23316 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:13.251780 23316 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:13.251852 23316 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:13.255146 23276 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: acquiring CA information for follower catalog manager: success
I20251023 20:15:13.257476 23276 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:15:13.279189 23288 catalog_manager.cc:1269] Loaded cluster ID: 67ec0cebd86e4322972b698542d64fab
I20251023 20:15:13.279456 23288 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: loading cluster ID for follower catalog manager: success
I20251023 20:15:13.284942 23288 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: acquiring CA information for follower catalog manager: success
I20251023 20:15:13.288925 23288 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:15:13.304003 23316 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.129:34961
I20251023 20:15:13.304180 23443 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.129:34961 every 8 connection(s)
I20251023 20:15:13.305660 23316 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/data/info.pb
I20251023 20:15:13.316169 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 23316
I20251023 20:15:13.316387 23038 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/instance
I20251023 20:15:13.321926 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.127.130:0
--local_ip_for_outbound_sockets=127.22.127.130
--webserver_interface=127.22.127.130
--webserver_port=0
--tserver_master_addrs=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251023 20:15:13.347527 23445 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
I20251023 20:15:13.348038 23445 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:13.349469 23445 heartbeater.cc:507] Master 127.22.127.190:34331 requested a full tablet report, sending...
I20251023 20:15:13.352336 23077 ts_manager.cc:194] Registered new tserver with Master: d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:13.353139 23444 heartbeater.cc:344] Connected to a master server at 127.22.127.188:41417
I20251023 20:15:13.353266 23444 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:13.353849 23444 heartbeater.cc:507] Master 127.22.127.188:41417 requested a full tablet report, sending...
I20251023 20:15:13.355522 23219 ts_manager.cc:194] Registered new tserver with Master: d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:13.358502 23446 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:13.358641 23446 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:13.359027 23446 heartbeater.cc:507] Master 127.22.127.189:33509 requested a full tablet report, sending...
I20251023 20:15:13.360337 23148 ts_manager.cc:194] Registered new tserver with Master: d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:13.361891 23148 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.127.129:60899
W20251023 20:15:13.682426 23452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:13.682706 23452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:13.682837 23452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:13.690848 23452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:13.691036 23452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.127.130
I20251023 20:15:13.701026 23452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.127.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.127.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.127.130
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
I20251023 20:15:13.702610 23452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:13.704319 23452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:13.714308 23460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:13.715226 23462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:13.715378 23459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:13.716491 23452 server_base.cc:1047] running on GCE node
I20251023 20:15:13.717181 23452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:13.718305 23452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:13.719532 23452 hybrid_clock.cc:648] HybridClock initialized: now 1761250513719465 us; error 54 us; skew 500 ppm
I20251023 20:15:13.719848 23452 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 20:15:13.722092 23452 webserver.cc:492] Webserver started at http://127.22.127.130:42435/ using document root <none> and password file <none>
I20251023 20:15:13.722812 23452 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:13.722898 23452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:13.723201 23452 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:13.725939 23452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data/instance:
uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
format_stamp: "Formatted at 2025-10-23 20:15:13 on dist-test-slave-g5bj"
I20251023 20:15:13.726593 23452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/instance:
uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
format_stamp: "Formatted at 2025-10-23 20:15:13 on dist-test-slave-g5bj"
I20251023 20:15:13.731678 23452 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251023 20:15:13.734841 23468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:13.736078 23452 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:13.736219 23452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal
uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
format_stamp: "Formatted at 2025-10-23 20:15:13 on dist-test-slave-g5bj"
I20251023 20:15:13.736387 23452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:13.752952 23452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:13.753767 23452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:13.754118 23452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:13.755435 23452 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:13.757053 23452 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:13.757164 23452 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251023 20:15:13.757261 23452 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:13.757326 23452 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:13.796489 23452 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.130:34173
I20251023 20:15:13.796541 23580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.130:34173 every 8 connection(s)
I20251023 20:15:13.797849 23452 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/data/info.pb
I20251023 20:15:13.806972 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 23452
I20251023 20:15:13.807175 23038 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/instance
I20251023 20:15:13.811044 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.127.131:0
--local_ip_for_outbound_sockets=127.22.127.131
--webserver_interface=127.22.127.131
--webserver_port=0
--tserver_master_addrs=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251023 20:15:13.815017 23582 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
I20251023 20:15:13.815493 23582 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:13.816463 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:13.816623 23583 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:13.816692 23582 heartbeater.cc:507] Master 127.22.127.190:34331 requested a full tablet report, sending...
I20251023 20:15:13.817152 23583 heartbeater.cc:507] Master 127.22.127.189:33509 requested a full tablet report, sending...
I20251023 20:15:13.818311 23148 ts_manager.cc:194] Registered new tserver with Master: ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:13.818897 23077 ts_manager.cc:194] Registered new tserver with Master: ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:13.819111 23148 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.127.130:34877
I20251023 20:15:13.820022 23581 heartbeater.cc:344] Connected to a master server at 127.22.127.188:41417
I20251023 20:15:13.820142 23581 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:13.820554 23581 heartbeater.cc:507] Master 127.22.127.188:41417 requested a full tablet report, sending...
I20251023 20:15:13.821513 23219 ts_manager.cc:194] Registered new tserver with Master: ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:13.864267 23446 heartbeater.cc:499] Master 127.22.127.189:33509 was elected leader, sending a full tablet report...
W20251023 20:15:14.156064 23588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:14.156363 23588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:14.156454 23588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251023 20:15:14.164906 23588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:14.165069 23588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.127.131
I20251023 20:15:14.175100 23588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.127.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.127.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.127.131
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
I20251023 20:15:14.176633 23588 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:14.178391 23588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251023 20:15:14.187369 23593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:14.187978 23596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:14.187999 23594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:14.188599 23588 server_base.cc:1047] running on GCE node
I20251023 20:15:14.189141 23588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:14.190007 23588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:14.191187 23588 hybrid_clock.cc:648] HybridClock initialized: now 1761250514191131 us; error 46 us; skew 500 ppm
I20251023 20:15:14.191453 23588 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 20:15:14.193464 23588 webserver.cc:492] Webserver started at http://127.22.127.131:32841/ using document root <none> and password file <none>
I20251023 20:15:14.194162 23588 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:14.194255 23588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:14.194594 23588 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251023 20:15:14.196823 23588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data/instance:
uuid: "9bc6522db11e4960a9aad84435371a32"
format_stamp: "Formatted at 2025-10-23 20:15:14 on dist-test-slave-g5bj"
I20251023 20:15:14.197638 23588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/instance:
uuid: "9bc6522db11e4960a9aad84435371a32"
format_stamp: "Formatted at 2025-10-23 20:15:14 on dist-test-slave-g5bj"
I20251023 20:15:14.202584 23588 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20251023 20:15:14.206193 23602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:14.207382 23588 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251023 20:15:14.207532 23588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal
uuid: "9bc6522db11e4960a9aad84435371a32"
format_stamp: "Formatted at 2025-10-23 20:15:14 on dist-test-slave-g5bj"
I20251023 20:15:14.207720 23588 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:14.225701 23588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:14.226536 23588 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:14.226774 23588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:14.227598 23588 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251023 20:15:14.229281 23588 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251023 20:15:14.229372 23588 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:14.229465 23588 ts_tablet_manager.cc:616] Registered 0 tablets
I20251023 20:15:14.229652 23588 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:14.265225 23588 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.131:46211
I20251023 20:15:14.265268 23714 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.131:46211 every 8 connection(s)
I20251023 20:15:14.266582 23588 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/data/info.pb
I20251023 20:15:14.271144 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 23588
I20251023 20:15:14.271301 23038 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/instance
I20251023 20:15:14.287199 23715 heartbeater.cc:344] Connected to a master server at 127.22.127.188:41417
I20251023 20:15:14.287235 23716 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
I20251023 20:15:14.287546 23715 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:14.287552 23716 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:14.288576 23716 heartbeater.cc:507] Master 127.22.127.190:34331 requested a full tablet report, sending...
I20251023 20:15:14.288602 23715 heartbeater.cc:507] Master 127.22.127.188:41417 requested a full tablet report, sending...
I20251023 20:15:14.290200 23077 ts_manager.cc:194] Registered new tserver with Master: 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:14.290621 23219 ts_manager.cc:194] Registered new tserver with Master: 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:14.291527 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:14.291633 23717 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:14.292011 23717 heartbeater.cc:507] Master 127.22.127.189:33509 requested a full tablet report, sending...
I20251023 20:15:14.292852 23148 ts_manager.cc:194] Registered new tserver with Master: 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:14.293762 23148 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.127.131:57147
I20251023 20:15:14.297699 23038 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251023 20:15:14.321350 23583 heartbeater.cc:499] Master 127.22.127.189:33509 was elected leader, sending a full tablet report...
I20251023 20:15:14.327267 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_bf9d6e8d268f41b28a4f568fbeb512c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.327190 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f297a4b70e214182a0c4af24347c8ab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.328770 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d0140542d63945abb01c4e316d490278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.329010 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f73e49c5980346d8b39cea3c2bffa739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.327522 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_43bf18200b234de68377af27f882f395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.331293 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0b9bd935bba8401c8990328116d7f600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.333559 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f297a4b70e214182a0c4af24347c8ab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.333560 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b9bd935bba8401c8990328116d7f600 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.333559 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf9d6e8d268f41b28a4f568fbeb512c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.333997 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43bf18200b234de68377af27f882f395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.331007 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8b24573ba4864a13a123e5d8d2bf9448"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.334715 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b24573ba4864a13a123e5d8d2bf9448 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.333799 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f73e49c5980346d8b39cea3c2bffa739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.337574 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0140542d63945abb01c4e316d490278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.326601 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c747a4f8ca7a461e90896f4145d9ceb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.340271 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c747a4f8ca7a461e90896f4145d9ceb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.344146 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0acacb2edf1c461baf55cc00e9a69ff6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.344532 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_50a809e112244d0984bf222990b09883"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.345042 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0acacb2edf1c461baf55cc00e9a69ff6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.345260 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50a809e112244d0984bf222990b09883 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.368855 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f297a4b70e214182a0c4af24347c8ab0" } new_table_name: "default.Table_bbb88a1064eb4a27aa71756ff68bb6db" modify_external_catalogs: true
I20251023 20:15:14.436036 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f73e49c5980346d8b39cea3c2bffa739" } new_table_name: "default.Table_93ffb17658084632a48221ae255f149d" modify_external_catalogs: true
W20251023 20:15:14.437099 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9ebd537f271426a95af432ec9070ae5 consensus state has no leader
I20251023 20:15:14.438211 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bf9d6e8d268f41b28a4f568fbeb512c0" } new_table_name: "default.Table_426958573170402182c7e797e37f1e44" modify_external_catalogs: true
I20251023 20:15:14.439535 23145 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d0140542d63945abb01c4e316d490278" } new_table_name: "default.Table_a217b86781b147cc9705ef222df5e930" modify_external_catalogs: true
I20251023 20:15:14.440883 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c747a4f8ca7a461e90896f4145d9ceb6" } modify_external_catalogs: true
I20251023 20:15:14.441190 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c747a4f8ca7a461e90896f4145d9ceb6" } modify_external_catalogs: true
I20251023 20:15:14.442364 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0acacb2edf1c461baf55cc00e9a69ff6" } modify_external_catalogs: true
I20251023 20:15:14.442559 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0acacb2edf1c461baf55cc00e9a69ff6" } modify_external_catalogs: true
I20251023 20:15:14.443109 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0b9bd935bba8401c8990328116d7f600" } new_table_name: "default.Table_7b7f45411bb14f6e8c811e72f6c92281" modify_external_catalogs: true
I20251023 20:15:14.443480 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8b24573ba4864a13a123e5d8d2bf9448" } modify_external_catalogs: true
I20251023 20:15:14.443864 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_43bf18200b234de68377af27f882f395" } modify_external_catalogs: true
I20251023 20:15:14.444026 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_43bf18200b234de68377af27f882f395" } modify_external_catalogs: true
I20251023 20:15:14.444372 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8b24573ba4864a13a123e5d8d2bf9448" } modify_external_catalogs: true
I20251023 20:15:14.444428 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_50a809e112244d0984bf222990b09883" } modify_external_catalogs: true
I20251023 20:15:14.444571 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_50a809e112244d0984bf222990b09883" } modify_external_catalogs: true
W20251023 20:15:14.455026 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcec61131b7f4af590acf88a2ab3c5f1 has no consensus state
W20251023 20:15:14.455487 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d05145bbb50343f38166de6b35bad890 has no consensus state
W20251023 20:15:14.456440 23145 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd41c9c84d814f9191d5952e71e31a06 has no consensus state
W20251023 20:15:14.457545 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b09c0412f1eb48df8de542ca687a7b48 has no consensus state
W20251023 20:15:14.484267 23135 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet fcec61131b7f4af590acf88a2ab3c5f1 (table default.Table_426958573170402182c7e797e37f1e44 [id=0d5215e658b64dbcbc9d37fc2622ca61], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fcec61131b7f4af590acf88a2ab3c5f1 has no consensus state
I20251023 20:15:14.488188 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 56fd87a212ff4ce893a44597ec038be9
I20251023 20:15:14.489385 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 536216374bae4bc481d78fabbed3567f
I20251023 20:15:14.490950 23302 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 536216374bae4bc481d78fabbed3567f
I20251023 20:15:14.491385 23302 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 56fd87a212ff4ce893a44597ec038be9
I20251023 20:15:14.493594 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 6ae1876fa334495ca95dc2e9e6b4a0fd
I20251023 20:15:14.493858 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 73f205f4e6d547229f3026c16a6fd485
I20251023 20:15:14.498535 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet eedc0251bae548bfb4f1dd54f33b7a86
I20251023 20:15:14.509382 23038 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu pid 23046
I20251023 20:15:14.510785 23107 generic_service.cc:196] Checking for leaks (request via RPC)
I20251023 20:15:14.516860 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.516904 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.519263 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.519619 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.520272 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 536216374bae4bc481d78fabbed3567f (DEFAULT_TABLE table=default.Table_50a809e112244d0984bf222990b09883 [id=5991f18468ca47ac82800d008795e0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.520273 23650 tablet_service.cc:1505] Processing CreateTablet for tablet d9ebd537f271426a95af432ec9070ae5 (DEFAULT_TABLE table=default.Table_bbb88a1064eb4a27aa71756ff68bb6db [id=197fa361f58a4b24b5b394eb7f43b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.521790 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 536216374bae4bc481d78fabbed3567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.521906 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.522071 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f205f4e6d547229f3026c16a6fd485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.522375 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 56fd87a212ff4ce893a44597ec038be9 (DEFAULT_TABLE table=default.Table_0acacb2edf1c461baf55cc00e9a69ff6 [id=8478b83c0ee645fba6c41151e56246b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.522665 23645 tablet_service.cc:1505] Processing CreateTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 (DEFAULT_TABLE table=default.Table_426958573170402182c7e797e37f1e44 [id=0d5215e658b64dbcbc9d37fc2622ca61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.522850 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd (DEFAULT_TABLE table=default.Table_8b24573ba4864a13a123e5d8d2bf9448 [id=fb7af3cee68a4f3c9307a5876db279c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.523725 23644 tablet_service.cc:1505] Processing CreateTablet for tablet d05145bbb50343f38166de6b35bad890 (DEFAULT_TABLE table=default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.523875 23643 tablet_service.cc:1505] Processing CreateTablet for tablet 73f205f4e6d547229f3026c16a6fd485 (DEFAULT_TABLE table=default.Table_43bf18200b234de68377af27f882f395 [id=da87b3d063a64aab92f939df02727144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.524106 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.524562 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f205f4e6d547229f3026c16a6fd485. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.524838 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcec61131b7f4af590acf88a2ab3c5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.525061 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ebd537f271426a95af432ec9070ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.525453 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05145bbb50343f38166de6b35bad890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.526329 23641 tablet_service.cc:1505] Processing CreateTablet for tablet bd41c9c84d814f9191d5952e71e31a06 (DEFAULT_TABLE table=default.Table_a217b86781b147cc9705ef222df5e930 [id=07cb3d69214745d3aa5003e92b2a09c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.526468 23642 tablet_service.cc:1505] Processing CreateTablet for tablet b09c0412f1eb48df8de542ca687a7b48 (DEFAULT_TABLE table=default.Table_7b7f45411bb14f6e8c811e72f6c92281 [id=18ee017a8d484c7b9ff7dbbceaa5f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.526641 23647 tablet_service.cc:1505] Processing CreateTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 (DEFAULT_TABLE table=default.Table_c747a4f8ca7a461e90896f4145d9ceb6 [id=08f1b14fc24d49baa3869370048151aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.527495 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd41c9c84d814f9191d5952e71e31a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.527818 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09c0412f1eb48df8de542ca687a7b48. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.528064 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 536216374bae4bc481d78fabbed3567f because the tablet was not found. No further retry: Not found: Tablet not found: 536216374bae4bc481d78fabbed3567f
W20251023 20:15:14.528491 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 56fd87a212ff4ce893a44597ec038be9 because the tablet was not found. No further retry: Not found: Tablet not found: 56fd87a212ff4ce893a44597ec038be9
W20251023 20:15:14.530820 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 536216374bae4bc481d78fabbed3567f because the tablet was not found. No further retry: Not found: Tablet not found: 536216374bae4bc481d78fabbed3567f
I20251023 20:15:14.531430 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.533138 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.534582 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.531872 23509 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f205f4e6d547229f3026c16a6fd485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.537371 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.539000 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.539260 23376 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.539535 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.534823 23512 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.541384 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fd87a212ff4ce893a44597ec038be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.541986 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.543324 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.544004 23376 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f205f4e6d547229f3026c16a6fd485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.546164 23379 tablet_service.cc:1505] Processing CreateTablet for tablet d9ebd537f271426a95af432ec9070ae5 (DEFAULT_TABLE table=default.Table_bbb88a1064eb4a27aa71756ff68bb6db [id=197fa361f58a4b24b5b394eb7f43b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.547830 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ebd537f271426a95af432ec9070ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.549225 23368 tablet_service.cc:1505] Processing CreateTablet for tablet b09c0412f1eb48df8de542ca687a7b48 (DEFAULT_TABLE table=default.Table_7b7f45411bb14f6e8c811e72f6c92281 [id=18ee017a8d484c7b9ff7dbbceaa5f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.549715 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09c0412f1eb48df8de542ca687a7b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.553248 23372 tablet_service.cc:1505] Processing CreateTablet for tablet d05145bbb50343f38166de6b35bad890 (DEFAULT_TABLE table=default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.553381 23367 tablet_service.cc:1505] Processing CreateTablet for tablet bd41c9c84d814f9191d5952e71e31a06 (DEFAULT_TABLE table=default.Table_a217b86781b147cc9705ef222df5e930 [id=07cb3d69214745d3aa5003e92b2a09c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.553896 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05145bbb50343f38166de6b35bad890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.557502 23373 tablet_service.cc:1505] Processing CreateTablet for tablet 56fd87a212ff4ce893a44597ec038be9 (DEFAULT_TABLE table=default.Table_0acacb2edf1c461baf55cc00e9a69ff6 [id=8478b83c0ee645fba6c41151e56246b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.557845 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 73f205f4e6d547229f3026c16a6fd485 (DEFAULT_TABLE table=default.Table_43bf18200b234de68377af27f882f395 [id=da87b3d063a64aab92f939df02727144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.553274 23369 tablet_service.cc:1505] Processing CreateTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 (DEFAULT_TABLE table=default.Table_c747a4f8ca7a461e90896f4145d9ceb6 [id=08f1b14fc24d49baa3869370048151aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.558290 23377 tablet_service.cc:1505] Processing CreateTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 (DEFAULT_TABLE table=default.Table_426958573170402182c7e797e37f1e44 [id=0d5215e658b64dbcbc9d37fc2622ca61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.558792 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fd87a212ff4ce893a44597ec038be9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.562436 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 56fd87a212ff4ce893a44597ec038be9 because the tablet was not found. No further retry: Not found: Tablet not found: 56fd87a212ff4ce893a44597ec038be9
I20251023 20:15:14.564771 23502 tablet_service.cc:1505] Processing CreateTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 (DEFAULT_TABLE table=default.Table_c747a4f8ca7a461e90896f4145d9ceb6 [id=08f1b14fc24d49baa3869370048151aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.566649 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eedc0251bae548bfb4f1dd54f33b7a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.567216 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 536216374bae4bc481d78fabbed3567f (DEFAULT_TABLE table=default.Table_50a809e112244d0984bf222990b09883 [id=5991f18468ca47ac82800d008795e0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.567894 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_57c800e050ac4e75a5cdbfff336ec344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.568467 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57c800e050ac4e75a5cdbfff336ec344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.569010 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd (DEFAULT_TABLE table=default.Table_8b24573ba4864a13a123e5d8d2bf9448 [id=fb7af3cee68a4f3c9307a5876db279c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.569821 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 536216374bae4bc481d78fabbed3567f (DEFAULT_TABLE table=default.Table_50a809e112244d0984bf222990b09883 [id=5991f18468ca47ac82800d008795e0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.570991 23504 tablet_service.cc:1505] Processing CreateTablet for tablet d05145bbb50343f38166de6b35bad890 (DEFAULT_TABLE table=default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.572607 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.573168 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 56fd87a212ff4ce893a44597ec038be9 (DEFAULT_TABLE table=default.Table_0acacb2edf1c461baf55cc00e9a69ff6 [id=8478b83c0ee645fba6c41151e56246b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.573930 23503 tablet_service.cc:1505] Processing CreateTablet for tablet 73f205f4e6d547229f3026c16a6fd485 (DEFAULT_TABLE table=default.Table_43bf18200b234de68377af27f882f395 [id=da87b3d063a64aab92f939df02727144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.574577 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.575712 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd (DEFAULT_TABLE table=default.Table_8b24573ba4864a13a123e5d8d2bf9448 [id=fb7af3cee68a4f3c9307a5876db279c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.577785 23501 tablet_service.cc:1505] Processing CreateTablet for tablet b09c0412f1eb48df8de542ca687a7b48 (DEFAULT_TABLE table=default.Table_7b7f45411bb14f6e8c811e72f6c92281 [id=18ee017a8d484c7b9ff7dbbceaa5f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.575961 23506 tablet_service.cc:1505] Processing CreateTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 (DEFAULT_TABLE table=default.Table_426958573170402182c7e797e37f1e44 [id=0d5215e658b64dbcbc9d37fc2622ca61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.578715 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a4c889bc3eb948a5988468904f855712"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.579244 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4c889bc3eb948a5988468904f855712 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.580485 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 56fd87a212ff4ce893a44597ec038be9 because the tablet was not found. No further retry: Not found: Tablet not found: 56fd87a212ff4ce893a44597ec038be9
W20251023 20:15:14.581405 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 536216374bae4bc481d78fabbed3567f because the tablet was not found. No further retry: Not found: Tablet not found: 536216374bae4bc481d78fabbed3567f
W20251023 20:15:14.581703 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 536216374bae4bc481d78fabbed3567f because the tablet was not found. No further retry: Not found: Tablet not found: 536216374bae4bc481d78fabbed3567f
W20251023 20:15:14.581910 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 73f205f4e6d547229f3026c16a6fd485 because the tablet was not found. No further retry: Not found: Tablet not found: 73f205f4e6d547229f3026c16a6fd485
W20251023 20:15:14.582106 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 56fd87a212ff4ce893a44597ec038be9 because the tablet was not found. No further retry: Not found: Tablet not found: 56fd87a212ff4ce893a44597ec038be9
W20251023 20:15:14.582283 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd because the tablet was not found. No further retry: Not found: Tablet not found: 6ae1876fa334495ca95dc2e9e6b4a0fd
W20251023 20:15:14.582477 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet eedc0251bae548bfb4f1dd54f33b7a86 because the tablet was not found. No further retry: Not found: Tablet not found: eedc0251bae548bfb4f1dd54f33b7a86
I20251023 20:15:14.584283 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f205f4e6d547229f3026c16a6fd485. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.584476 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 536216374bae4bc481d78fabbed3567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.584656 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56fd87a212ff4ce893a44597ec038be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.584794 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05145bbb50343f38166de6b35bad890. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.584985 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcec61131b7f4af590acf88a2ab3c5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.585161 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.584939 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_89bfd4f7096f41728d0c80c53f1f4268"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.585309 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09c0412f1eb48df8de542ca687a7b48. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.585628 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89bfd4f7096f41728d0c80c53f1f4268 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.589321 23505 tablet_service.cc:1505] Processing CreateTablet for tablet e87648998c0b4297ab644d2c39ab2132 (DEFAULT_TABLE table=default.Table_57c800e050ac4e75a5cdbfff336ec344 [id=823681dedfc3477d995ad3695c4a9541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.589903 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87648998c0b4297ab644d2c39ab2132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.592989 23500 tablet_service.cc:1505] Processing CreateTablet for tablet bd41c9c84d814f9191d5952e71e31a06 (DEFAULT_TABLE table=default.Table_a217b86781b147cc9705ef222df5e930 [id=07cb3d69214745d3aa5003e92b2a09c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.593771 23500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd41c9c84d814f9191d5952e71e31a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.593284 23516 tablet_service.cc:1505] Processing CreateTablet for tablet d9ebd537f271426a95af432ec9070ae5 (DEFAULT_TABLE table=default.Table_bbb88a1064eb4a27aa71756ff68bb6db [id=197fa361f58a4b24b5b394eb7f43b4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.594343 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ebd537f271426a95af432ec9070ae5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.596623 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd because the tablet was not found. No further retry: Not found: Tablet not found: 6ae1876fa334495ca95dc2e9e6b4a0fd
W20251023 20:15:14.602875 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 73f205f4e6d547229f3026c16a6fd485 because the tablet was not found. No further retry: Not found: Tablet not found: 73f205f4e6d547229f3026c16a6fd485
W20251023 20:15:14.603161 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet eedc0251bae548bfb4f1dd54f33b7a86 because the tablet was not found. No further retry: Not found: Tablet not found: eedc0251bae548bfb4f1dd54f33b7a86
I20251023 20:15:14.603658 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 2631175754104e4a9821849f27f04ecb (DEFAULT_TABLE table=default.Table_a4c889bc3eb948a5988468904f855712 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.614809 23640 tablet_service.cc:1505] Processing CreateTablet for tablet e87648998c0b4297ab644d2c39ab2132 (DEFAULT_TABLE table=default.Table_57c800e050ac4e75a5cdbfff336ec344 [id=823681dedfc3477d995ad3695c4a9541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.615116 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 (DEFAULT_TABLE table=default.Table_89bfd4f7096f41728d0c80c53f1f4268 [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.615181 23638 tablet_service.cc:1505] Processing CreateTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 (DEFAULT_TABLE table=default.Table_89bfd4f7096f41728d0c80c53f1f4268 [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.615694 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2d1315afa0468c956adf64e1bd9cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.616580 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 2631175754104e4a9821849f27f04ecb (DEFAULT_TABLE table=default.Table_a4c889bc3eb948a5988468904f855712 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.617131 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2631175754104e4a9821849f27f04ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.620519 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.622364 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eedc0251bae548bfb4f1dd54f33b7a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.623445 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87648998c0b4297ab644d2c39ab2132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.624217 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2631175754104e4a9821849f27f04ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.626195 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_30287d97a531423da0a359bc1eb053da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.626794 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30287d97a531423da0a359bc1eb053da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.628046 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d94baddda6104b5c9c007c95623ea0dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.628587 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d94baddda6104b5c9c007c95623ea0dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.629704 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_da0691e8e6224cfebf17fe7df75954dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.630141 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da0691e8e6224cfebf17fe7df75954dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.634590 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8e8a728158404934a7b077db56620986"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.635056 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e8a728158404934a7b077db56620986 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.635656 23638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2d1315afa0468c956adf64e1bd9cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.636070 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_14d3b90b6a8a4de7bfa30d4cc868869a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.636488 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14d3b90b6a8a4de7bfa30d4cc868869a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.637533 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.637933 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.638876 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_351eb0009756472096af129cb30c34e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.639256 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_351eb0009756472096af129cb30c34e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.646063 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7b7f45411bb14f6e8c811e72f6c92281" } new_table_name: "default.Table_45640565c4a24ff88274bc4e62b865ff" modify_external_catalogs: true
I20251023 20:15:14.658320 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet 2631175754104e4a9821849f27f04ecb from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:14.658675 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a2d1315afa0468c956adf64e1bd9cf0 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:14.659128 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a2d1315afa0468c956adf64e1bd9cf0 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:14.659436 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a2d1315afa0468c956adf64e1bd9cf0 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:14.665360 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 2631175754104e4a9821849f27f04ecb from {username='slave'} at 127.0.0.1:40384
W20251023 20:15:14.674850 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b09c0412f1eb48df8de542ca687a7b48 consensus state has no leader
I20251023 20:15:14.678473 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_426958573170402182c7e797e37f1e44" } new_table_name: "default.Table_07bcc06b3cdd42379508c3f649756c40" modify_external_catalogs: true
I20251023 20:15:14.678579 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_351eb0009756472096af129cb30c34e2" } new_table_name: "default.Table_e81dbc08f93d42d18c38eccb996633d4" modify_external_catalogs: true
I20251023 20:15:14.680037 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_14d3b90b6a8a4de7bfa30d4cc868869a" } new_table_name: "default.Table_f66ae376b73c46d180bde83fb80dc7bb" modify_external_catalogs: true
I20251023 20:15:14.680847 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a217b86781b147cc9705ef222df5e930" } new_table_name: "default.Table_8aa752a341ce41b896d76afaf6a0d4f8" modify_external_catalogs: true
W20251023 20:15:14.687697 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b125422cf17145d5aef418817082eb32 has no consensus state
W20251023 20:15:14.688761 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcec61131b7f4af590acf88a2ab3c5f1 consensus state has no leader
I20251023 20:15:14.688932 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 (DEFAULT_TABLE table=default.Table_30287d97a531423da0a359bc1eb053da [id=6e38fec2eebd48b8a6364be16ec99997]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:14.689303 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd41c9c84d814f9191d5952e71e31a06 consensus state has no leader
I20251023 20:15:14.689350 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.696082 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9b25166b00842948980cdfcacdd5df2 has no consensus state
I20251023 20:15:14.700536 23512 tablet_service.cc:1505] Processing CreateTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 (DEFAULT_TABLE table=default.Table_8e8a728158404934a7b077db56620986 [id=a7ccdb7723b44991be2809bf1d80d6a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.701165 23512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091e54a94f2d4c40a0006c7e35f74823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.705404 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bbb88a1064eb4a27aa71756ff68bb6db" } modify_external_catalogs: true
I20251023 20:15:14.705565 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_57c800e050ac4e75a5cdbfff336ec344" } modify_external_catalogs: true
I20251023 20:15:14.705659 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bbb88a1064eb4a27aa71756ff68bb6db" } modify_external_catalogs: true
I20251023 20:15:14.705756 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_57c800e050ac4e75a5cdbfff336ec344" } modify_external_catalogs: true
I20251023 20:15:14.706662 23513 tablet_service.cc:1505] Processing CreateTablet for tablet dd456675077a4c73933c26e9a0313848 (DEFAULT_TABLE table=default.Table_da0691e8e6224cfebf17fe7df75954dd [id=a9e1acb22d6f423f924ad7417f9a986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.706961 23145 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_07bcc06b3cdd42379508c3f649756c40" } modify_external_catalogs: true
I20251023 20:15:14.707136 23513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd456675077a4c73933c26e9a0313848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.707101 23145 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_07bcc06b3cdd42379508c3f649756c40" } modify_external_catalogs: true
I20251023 20:15:14.708164 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3" } modify_external_catalogs: true
I20251023 20:15:14.708323 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3" } modify_external_catalogs: true
I20251023 20:15:14.707101 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_da0691e8e6224cfebf17fe7df75954dd" } modify_external_catalogs: true
I20251023 20:15:14.708518 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_da0691e8e6224cfebf17fe7df75954dd" } modify_external_catalogs: true
I20251023 20:15:14.709398 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 2177416169ca4d0d8368b1cac90c513e (DEFAULT_TABLE table=default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3 [id=2d40a1f71ab847df9127c80fd25f05af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.710363 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2177416169ca4d0d8368b1cac90c513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.710920 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 2631175754104e4a9821849f27f04ecb
I20251023 20:15:14.711369 23140 catalog_manager.cc:6719] ReplaceTablet: tablet 2631175754104e4a9821849f27f04ecb deleted and replaced by tablet c3035573f53c4a6db0bb72882ca84f37
I20251023 20:15:14.712070 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:14.712973 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2631175754104e4a9821849f27f04ecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2631175754104e4a9821849f27f04ecb already in progress: creating tablet
I20251023 20:15:14.713500 23499 tablet_service.cc:1505] Processing CreateTablet for tablet b125422cf17145d5aef418817082eb32 (DEFAULT_TABLE table=default.Table_e81dbc08f93d42d18c38eccb996633d4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.714092 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125422cf17145d5aef418817082eb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.717468 23509 tablet_service.cc:1505] Processing CreateTablet for tablet d9b25166b00842948980cdfcacdd5df2 (DEFAULT_TABLE table=default.Table_f66ae376b73c46d180bde83fb80dc7bb [id=4f40834f1fae488bb070b0c40c191ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.720206 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 9a2d1315afa0468c956adf64e1bd9cf0
I20251023 20:15:14.720468 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b25166b00842948980cdfcacdd5df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.720678 23143 catalog_manager.cc:6719] ReplaceTablet: tablet 9a2d1315afa0468c956adf64e1bd9cf0 deleted and replaced by tablet 24528de9d0a84eddb091dec4f1d2b4e2
I20251023 20:15:14.722009 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 24528de9d0a84eddb091dec4f1d2b4e2) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:14.722723 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a2d1315afa0468c956adf64e1bd9cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2d1315afa0468c956adf64e1bd9cf0 already in progress: creating tablet
I20251023 20:15:14.723241 23376 tablet_service.cc:1505] Processing CreateTablet for tablet e87648998c0b4297ab644d2c39ab2132 (DEFAULT_TABLE table=default.Table_57c800e050ac4e75a5cdbfff336ec344 [id=823681dedfc3477d995ad3695c4a9541]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.729282 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f205f4e6d547229f3026c16a6fd485. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.730156 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d9ebd537f271426a95af432ec9070ae5
I20251023 20:15:14.730588 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eedc0251bae548bfb4f1dd54f33b7a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.731902 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 536216374bae4bc481d78fabbed3567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.733045 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet d9ebd537f271426a95af432ec9070ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.733459 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.734970 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9ebd537f271426a95af432ec9070ae5 because the tablet was not found. No further retry: Not found: Tablet not found: d9ebd537f271426a95af432ec9070ae5
I20251023 20:15:14.736857 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet e87648998c0b4297ab644d2c39ab2132
I20251023 20:15:14.739456 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd41c9c84d814f9191d5952e71e31a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.740002 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet fcec61131b7f4af590acf88a2ab3c5f1
I20251023 20:15:14.740485 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet e87648998c0b4297ab644d2c39ab2132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:14.741839 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e87648998c0b4297ab644d2c39ab2132 because the tablet was not found. No further retry: Not found: Tablet not found: e87648998c0b4297ab644d2c39ab2132
I20251023 20:15:14.742151 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:14.743162 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fcec61131b7f4af590acf88a2ab3c5f1 because the tablet was not found. No further retry: Not found: Tablet not found: fcec61131b7f4af590acf88a2ab3c5f1
I20251023 20:15:14.743716 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet dd456675077a4c73933c26e9a0313848
I20251023 20:15:14.744586 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87648998c0b4297ab644d2c39ab2132. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.744719 23512 tablet_service.cc:1552] Processing DeleteTablet for tablet dd456675077a4c73933c26e9a0313848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:14.746433 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet dd456675077a4c73933c26e9a0313848 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd456675077a4c73933c26e9a0313848 already in progress: creating tablet
I20251023 20:15:14.742926 23364 tablet_service.cc:1505] Processing CreateTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 (DEFAULT_TABLE table=default.Table_8e8a728158404934a7b077db56620986 [id=a7ccdb7723b44991be2809bf1d80d6a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.739607 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 2631175754104e4a9821849f27f04ecb (DEFAULT_TABLE table=default.Table_a4c889bc3eb948a5988468904f855712 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.743674 23362 tablet_service.cc:1505] Processing CreateTablet for tablet dd456675077a4c73933c26e9a0313848 (DEFAULT_TABLE table=default.Table_da0691e8e6224cfebf17fe7df75954dd [id=a9e1acb22d6f423f924ad7417f9a986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.744004 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 2177416169ca4d0d8368b1cac90c513e (DEFAULT_TABLE table=default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3 [id=2d40a1f71ab847df9127c80fd25f05af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.744315 23360 tablet_service.cc:1505] Processing CreateTablet for tablet b125422cf17145d5aef418817082eb32 (DEFAULT_TABLE table=default.Table_e81dbc08f93d42d18c38eccb996633d4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.743346 23363 tablet_service.cc:1505] Processing CreateTablet for tablet d9b25166b00842948980cdfcacdd5df2 (DEFAULT_TABLE table=default.Table_f66ae376b73c46d180bde83fb80dc7bb [id=4f40834f1fae488bb070b0c40c191ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.745188 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 (DEFAULT_TABLE table=default.Table_30287d97a531423da0a359bc1eb053da [id=6e38fec2eebd48b8a6364be16ec99997]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.748080 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 2177416169ca4d0d8368b1cac90c513e
I20251023 20:15:14.748261 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 (DEFAULT_TABLE table=default.Table_89bfd4f7096f41728d0c80c53f1f4268 [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.750851 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcec61131b7f4af590acf88a2ab3c5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.751211 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 2177416169ca4d0d8368b1cac90c513e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:14.751585 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 (DEFAULT_TABLE table=default.Table_d94baddda6104b5c9c007c95623ea0dd [id=ce02fa77823e4fbcad36a7e21bf10630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.751948 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 216e2c887a364c8ea5068c3834d1e4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.752027 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.753190 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b25166b00842948980cdfcacdd5df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.754235 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd456675077a4c73933c26e9a0313848. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.755255 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2177416169ca4d0d8368b1cac90c513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.756250 23360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125422cf17145d5aef418817082eb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.757287 23364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091e54a94f2d4c40a0006c7e35f74823. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.757661 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2177416169ca4d0d8368b1cac90c513e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2177416169ca4d0d8368b1cac90c513e already in progress: creating tablet
I20251023 20:15:14.758414 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2631175754104e4a9821849f27f04ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.759486 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2d1315afa0468c956adf64e1bd9cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.760747 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 (DEFAULT_TABLE table=default.Table_89bfd4f7096f41728d0c80c53f1f4268 [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.761138 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24528de9d0a84eddb091dec4f1d2b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.768019 23516 tablet_service.cc:1505] Processing CreateTablet for tablet c3035573f53c4a6db0bb72882ca84f37 (DEFAULT_TABLE table=default.Table_a4c889bc3eb948a5988468904f855712 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.768545 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3035573f53c4a6db0bb72882ca84f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.816864 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b28a0fe65fc64b83b50ffe933aca75f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.817611 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b28a0fe65fc64b83b50ffe933aca75f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.819031 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_579a7389d3174efdbaf6d1724548bf04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.819552 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_579a7389d3174efdbaf6d1724548bf04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.822525 23717 heartbeater.cc:499] Master 127.22.127.189:33509 was elected leader, sending a full tablet report...
I20251023 20:15:14.826640 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.827036 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 24528de9d0a84eddb091dec4f1d2b4e2) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.827168 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 (DEFAULT_TABLE table=default.Table_30287d97a531423da0a359bc1eb053da [id=6e38fec2eebd48b8a6364be16ec99997]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.827294 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet d9ebd537f271426a95af432ec9070ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.827428 23787 tablet_bootstrap.cc:492] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:14.827471 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet e87648998c0b4297ab644d2c39ab2132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.827608 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.827626 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.827772 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet dd456675077a4c73933c26e9a0313848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.828896 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d592564ff9cd44ab86010457642460ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.829031 23650 tablet_service.cc:1505] Processing CreateTablet for tablet dd456675077a4c73933c26e9a0313848 (DEFAULT_TABLE table=default.Table_da0691e8e6224cfebf17fe7df75954dd [id=a9e1acb22d6f423f924ad7417f9a986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.829412 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd456675077a4c73933c26e9a0313848. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.829456 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d592564ff9cd44ab86010457642460ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.830297 23648 tablet_service.cc:1505] Processing CreateTablet for tablet d9b25166b00842948980cdfcacdd5df2 (DEFAULT_TABLE table=default.Table_f66ae376b73c46d180bde83fb80dc7bb [id=4f40834f1fae488bb070b0c40c191ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.830648 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b25166b00842948980cdfcacdd5df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.832691 23038 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu with pid 23046
I20251023 20:15:14.834798 23787 tablet_bootstrap.cc:654] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:14.835319 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 (DEFAULT_TABLE table=default.Table_d94baddda6104b5c9c007c95623ea0dd [id=ce02fa77823e4fbcad36a7e21bf10630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.836197 23787 log.cc:826] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:14.841786 23507 tablet_service.cc:1505] Processing CreateTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb (DEFAULT_TABLE table=default.Table_b28a0fe65fc64b83b50ffe933aca75f9 [id=fc58c27a539f4454acbba8fbf3b62461]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.842252 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ad21e3888a4915ae5f252c2fca64cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.852156 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 (DEFAULT_TABLE table=default.Table_579a7389d3174efdbaf6d1724548bf04 [id=f96dd558958e41a5a9dbf10177fe1277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.852653 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.853390 23786 tablet_bootstrap.cc:492] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:14.853401 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 (DEFAULT_TABLE table=default.Table_8e8a728158404934a7b077db56620986 [id=a7ccdb7723b44991be2809bf1d80d6a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.857772 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091e54a94f2d4c40a0006c7e35f74823. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.858388 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 0d73da547fd04557abd5aba73e76dd18 (DEFAULT_TABLE table=default.Table_d592564ff9cd44ab86010457642460ec [id=8d6a2964df664438b8680d81e4f040aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.858762 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d73da547fd04557abd5aba73e76dd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.860368 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_56862b3e207f468ab903cbb43238bc2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.861042 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56862b3e207f468ab903cbb43238bc2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.869288 23641 tablet_service.cc:1505] Processing CreateTablet for tablet b125422cf17145d5aef418817082eb32 (DEFAULT_TABLE table=default.Table_e81dbc08f93d42d18c38eccb996633d4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.869760 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125422cf17145d5aef418817082eb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.873528 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet 2177416169ca4d0d8368b1cac90c513e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:14.874372 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.875170 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_344bdb8a8edd4641b4e9786c5d6ce520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.875854 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_344bdb8a8edd4641b4e9786c5d6ce520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.877190 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet d9ebd537f271426a95af432ec9070ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.877594 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 24528de9d0a84eddb091dec4f1d2b4e2) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.878598 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet e87648998c0b4297ab644d2c39ab2132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.878602 23365 tablet_service.cc:1505] Processing CreateTablet for tablet c3035573f53c4a6db0bb72882ca84f37 (DEFAULT_TABLE table=default.Table_a4c889bc3eb948a5988468904f855712 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.878966 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 216e2c887a364c8ea5068c3834d1e4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.879546 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a90cbb8972924d6aad10abd0327c9874"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.879827 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_212585028f354c3e9403737b7ea1c732"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.880151 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a90cbb8972924d6aad10abd0327c9874 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.880314 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_212585028f354c3e9403737b7ea1c732 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.883095 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_156963dcf5454475a0cae4d52eff17f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:14.883208 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4f168f47af364b3db52fbbc64de90a6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.883630 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_156963dcf5454475a0cae4d52eff17f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.883849 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_bd046e3e4fd9439c8aabbcee605207e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:14.884258 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd046e3e4fd9439c8aabbcee605207e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:14.885319 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_30287d97a531423da0a359bc1eb053da" } new_table_name: "default.Table_877570fc930d464e8348f118f840439c" modify_external_catalogs: true
W20251023 20:15:14.883657 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f168f47af364b3db52fbbc64de90a6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:14.889869 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2631175754104e4a9821849f27f04ecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2631175754104e4a9821849f27f04ecb already in progress: creating tablet
W20251023 20:15:14.890290 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9ebd537f271426a95af432ec9070ae5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ebd537f271426a95af432ec9070ae5 already in progress: creating tablet
I20251023 20:15:14.890265 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 (DEFAULT_TABLE table=default.Table_579a7389d3174efdbaf6d1724548bf04 [id=f96dd558958e41a5a9dbf10177fe1277]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:14.890488 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9a2d1315afa0468c956adf64e1bd9cf0 because the tablet was not found. No further retry: Not found: Tablet not found: 9a2d1315afa0468c956adf64e1bd9cf0
W20251023 20:15:14.890797 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e87648998c0b4297ab644d2c39ab2132 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e87648998c0b4297ab644d2c39ab2132 already in progress: creating tablet
I20251023 20:15:14.894759 23640 tablet_service.cc:1505] Processing CreateTablet for tablet c3035573f53c4a6db0bb72882ca84f37 (DEFAULT_TABLE table=default.Table_a4c889bc3eb948a5988468904f855712 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.897367 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 (DEFAULT_TABLE table=default.Table_89bfd4f7096f41728d0c80c53f1f4268 [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.899868 23142 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet d05145bbb50343f38166de6b35bad890 (table default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]). Expected version 1 got 0
I20251023 20:15:14.900538 23142 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b09c0412f1eb48df8de542ca687a7b48 (table default.Table_45640565c4a24ff88274bc4e62b865ff [id=18ee017a8d484c7b9ff7dbbceaa5f7ac]). Expected version 2 got 0
I20251023 20:15:14.900813 23142 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet bd41c9c84d814f9191d5952e71e31a06 (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:14.901065 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3035573f53c4a6db0bb72882ca84f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.901808 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:14.903883 23364 tablet_service.cc:1505] Processing CreateTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb (DEFAULT_TABLE table=default.Table_b28a0fe65fc64b83b50ffe933aca75f9 [id=fc58c27a539f4454acbba8fbf3b62461]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.905678 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b28a0fe65fc64b83b50ffe933aca75f9" } new_table_name: "default.Table_57a2a81ca5484113b5e2a0857d40dbbb" modify_external_catalogs: true
I20251023 20:15:14.907159 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 (DEFAULT_TABLE table=default.Table_579a7389d3174efdbaf6d1724548bf04 [id=f96dd558958e41a5a9dbf10177fe1277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.907708 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_89bfd4f7096f41728d0c80c53f1f4268" } new_table_name: "default.Table_46ae5f307de94522a026332b6aec3650" modify_external_catalogs: true
I20251023 20:15:14.909366 23786 tablet_bootstrap.cc:654] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:14.910650 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_56862b3e207f468ab903cbb43238bc2c" } new_table_name: "default.Table_c9b696fd225e430195d01e0efc28f219" modify_external_catalogs: true
I20251023 20:15:14.911701 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24528de9d0a84eddb091dec4f1d2b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.913151 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 0232c33db59641b3af27bfd15cd7fb81 (DEFAULT_TABLE table=default.Table_56862b3e207f468ab903cbb43238bc2c [id=78fe983663c94eabab93ee902bde6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.913661 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0232c33db59641b3af27bfd15cd7fb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.914896 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 0d73da547fd04557abd5aba73e76dd18 (DEFAULT_TABLE table=default.Table_d592564ff9cd44ab86010457642460ec [id=8d6a2964df664438b8680d81e4f040aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.918941 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.920697 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 0232c33db59641b3af27bfd15cd7fb81 (DEFAULT_TABLE table=default.Table_56862b3e207f468ab903cbb43238bc2c [id=78fe983663c94eabab93ee902bde6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:14.921110 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0232c33db59641b3af27bfd15cd7fb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:14.921797 23786 log.cc:826] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:14.922712 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d73da547fd04557abd5aba73e76dd18. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:14.924424 23582 heartbeater.cc:646] Failed to heartbeat to 127.22.127.190:34331 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.22.127.190:34331 (error 108)
W20251023 20:15:14.945845 23134 consensus_peers.cc:597] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 -> Peer 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331): Couldn't send request to peer 6545140d94b7485aa81c51ad233355cd. Status: Network error: Client connection negotiation failed: client connection to 127.22.127.190:34331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:14.947306 23334 connection.cc:537] client connection to 127.22.127.190:34331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:14.948505 23445 heartbeater.cc:646] Failed to heartbeat to 127.22.127.190:34331 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:14.955981 23786 tablet_bootstrap.cc:492] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:14.956388 23786 ts_tablet_manager.cc:1403] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.104s user 0.011s sys 0.004s
I20251023 20:15:14.970176 23786 raft_consensus.cc:359] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:14.972239 23786 raft_consensus.cc:385] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:14.972610 23786 raft_consensus.cc:740] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:14.974005 23786 consensus_queue.cc:260] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:14.983471 23786 ts_tablet_manager.cc:1434] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.027s user 0.008s sys 0.004s
I20251023 20:15:14.984537 23364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ad21e3888a4915ae5f252c2fca64cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.005981 23786 tablet_bootstrap.cc:492] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.007148 23647 tablet_service.cc:1505] Processing CreateTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb (DEFAULT_TABLE table=default.Table_b28a0fe65fc64b83b50ffe933aca75f9 [id=fc58c27a539f4454acbba8fbf3b62461]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.007547 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 2177416169ca4d0d8368b1cac90c513e (DEFAULT_TABLE table=default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3 [id=2d40a1f71ab847df9127c80fd25f05af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.007745 23790 tablet_bootstrap.cc:492] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:14.902047 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 (DEFAULT_TABLE table=default.Table_89bfd4f7096f41728d0c80c53f1f4268 [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.011008 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.011298 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3035573f53c4a6db0bb72882ca84f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.011381 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 0d73da547fd04557abd5aba73e76dd18 (DEFAULT_TABLE table=default.Table_d592564ff9cd44ab86010457642460ec [id=8d6a2964df664438b8680d81e4f040aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.011646 23790 tablet_bootstrap.cc:654] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.011703 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d73da547fd04557abd5aba73e76dd18. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.014135 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 56fd87a212ff4ce893a44597ec038be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56fd87a212ff4ce893a44597ec038be9 already in progress: creating tablet
W20251023 20:15:15.014781 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 536216374bae4bc481d78fabbed3567f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 536216374bae4bc481d78fabbed3567f already in progress: creating tablet
W20251023 20:15:15.015349 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 536216374bae4bc481d78fabbed3567f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 536216374bae4bc481d78fabbed3567f already in progress: creating tablet
W20251023 20:15:15.015902 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 56fd87a212ff4ce893a44597ec038be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56fd87a212ff4ce893a44597ec038be9 already in progress: creating tablet
W20251023 20:15:15.016397 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ae1876fa334495ca95dc2e9e6b4a0fd already in progress: creating tablet
W20251023 20:15:15.016909 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet eedc0251bae548bfb4f1dd54f33b7a86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eedc0251bae548bfb4f1dd54f33b7a86 already in progress: creating tablet
I20251023 20:15:15.017159 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet dd456675077a4c73933c26e9a0313848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.017431 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 73f205f4e6d547229f3026c16a6fd485 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73f205f4e6d547229f3026c16a6fd485 already in progress: creating tablet
I20251023 20:15:15.017583 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 (DEFAULT_TABLE table=default.Table_d94baddda6104b5c9c007c95623ea0dd [id=ce02fa77823e4fbcad36a7e21bf10630]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.017976 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 216e2c887a364c8ea5068c3834d1e4e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.017992 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2631175754104e4a9821849f27f04ecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2631175754104e4a9821849f27f04ecb already in progress: creating tablet
W20251023 20:15:15.018464 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9a2d1315afa0468c956adf64e1bd9cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2d1315afa0468c956adf64e1bd9cf0 already in progress: creating tablet
W20251023 20:15:15.018997 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d9ebd537f271426a95af432ec9070ae5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ebd537f271426a95af432ec9070ae5 already in progress: creating tablet
W20251023 20:15:15.019482 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e87648998c0b4297ab644d2c39ab2132 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e87648998c0b4297ab644d2c39ab2132 already in progress: creating tablet
W20251023 20:15:15.019999 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet fcec61131b7f4af590acf88a2ab3c5f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcec61131b7f4af590acf88a2ab3c5f1 already in progress: creating tablet
W20251023 20:15:15.020515 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet dd456675077a4c73933c26e9a0313848 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd456675077a4c73933c26e9a0313848 already in progress: creating tablet
W20251023 20:15:15.021124 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2177416169ca4d0d8368b1cac90c513e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2177416169ca4d0d8368b1cac90c513e already in progress: creating tablet
I20251023 20:15:15.023480 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet e87648998c0b4297ab644d2c39ab2132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.023725 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2177416169ca4d0d8368b1cac90c513e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.024284 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.024477 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.025451 23303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:15.025662 23303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:15.026010 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24528de9d0a84eddb091dec4f1d2b4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.026468 23786 tablet_bootstrap.cc:654] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.027307 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 2177416169ca4d0d8368b1cac90c513e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.027527 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.027945 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ad21e3888a4915ae5f252c2fca64cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.028796 23786 tablet_bootstrap.cc:492] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.028939 23786 ts_tablet_manager.cc:1403] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20251023 20:15:15.029929 23786 raft_consensus.cc:359] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.030194 23786 raft_consensus.cc:385] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.030326 23786 raft_consensus.cc:740] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.024830 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 0232c33db59641b3af27bfd15cd7fb81 (DEFAULT_TABLE table=default.Table_56862b3e207f468ab903cbb43238bc2c [id=78fe983663c94eabab93ee902bde6998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.030653 23786 consensus_queue.cc:260] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.033978 23786 ts_tablet_manager.cc:1434] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251023 20:15:15.034176 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a4c889bc3eb948a5988468904f855712" } new_table_name: "default.Table_2741dc8a0a2845cc945ac27e11d1bb98" modify_external_catalogs: true
I20251023 20:15:15.035422 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet bd41c9c84d814f9191d5952e71e31a06 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.035926 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.036697 23786 tablet_bootstrap.cc:492] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.038578 23786 tablet_bootstrap.cc:654] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:15.038812 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fcec61131b7f4af590acf88a2ab3c5f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcec61131b7f4af590acf88a2ab3c5f1 already in progress: creating tablet
W20251023 20:15:15.039129 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet dd456675077a4c73933c26e9a0313848 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd456675077a4c73933c26e9a0313848 already in progress: creating tablet
W20251023 20:15:15.039351 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e87648998c0b4297ab644d2c39ab2132 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e87648998c0b4297ab644d2c39ab2132 already in progress: creating tablet
W20251023 20:15:15.039538 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ae1876fa334495ca95dc2e9e6b4a0fd already in progress: creating tablet
W20251023 20:15:15.039819 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 536216374bae4bc481d78fabbed3567f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 536216374bae4bc481d78fabbed3567f already in progress: creating tablet
W20251023 20:15:15.039985 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2177416169ca4d0d8368b1cac90c513e because the tablet was not found. No further retry: Not found: Tablet not found: 2177416169ca4d0d8368b1cac90c513e
I20251023 20:15:15.040343 23786 tablet_bootstrap.cc:492] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.040485 23786 ts_tablet_manager.cc:1403] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:15.040758 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet d9ebd537f271426a95af432ec9070ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.041286 23605 connection.cc:537] client connection to 127.22.127.190:34331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:15.041285 23786 raft_consensus.cc:359] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.041561 23786 raft_consensus.cc:385] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.041644 23786 raft_consensus.cc:740] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.041857 23786 consensus_queue.cc:260] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.041992 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0232c33db59641b3af27bfd15cd7fb81. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.042560 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 56fd87a212ff4ce893a44597ec038be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56fd87a212ff4ce893a44597ec038be9 already in progress: creating tablet
I20251023 20:15:15.042814 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.043567 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9ebd537f271426a95af432ec9070ae5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ebd537f271426a95af432ec9070ae5 already in progress: creating tablet
I20251023 20:15:15.043960 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet b125422cf17145d5aef418817082eb32 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.044986 23786 ts_tablet_manager.cc:1434] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251023 20:15:15.045590 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2631175754104e4a9821849f27f04ecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2631175754104e4a9821849f27f04ecb already in progress: creating tablet
I20251023 20:15:15.045985 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.046692 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8ad21e3888a4915ae5f252c2fca64cb consensus state has no leader
W20251023 20:15:15.046859 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet eedc0251bae548bfb4f1dd54f33b7a86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eedc0251bae548bfb4f1dd54f33b7a86 already in progress: creating tablet
I20251023 20:15:15.046998 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet c3035573f53c4a6db0bb72882ca84f37 from {username='slave'} at 127.0.0.1:40384
W20251023 20:15:15.043541 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 consensus state has no leader
I20251023 20:15:15.047348 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet dd456675077a4c73933c26e9a0313848 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.047632 23786 tablet_bootstrap.cc:492] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.047888 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d94baddda6104b5c9c007c95623ea0dd" } modify_external_catalogs: true
I20251023 20:15:15.048025 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d94baddda6104b5c9c007c95623ea0dd" } modify_external_catalogs: true
W20251023 20:15:15.048077 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24528de9d0a84eddb091dec4f1d2b4e2 consensus state has no leader
W20251023 20:15:15.047358 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0232c33db59641b3af27bfd15cd7fb81 consensus state has no leader
I20251023 20:15:15.048398 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8e8a728158404934a7b077db56620986" } modify_external_catalogs: true
I20251023 20:15:15.048607 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f66ae376b73c46d180bde83fb80dc7bb" } modify_external_catalogs: true
I20251023 20:15:15.048633 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8e8a728158404934a7b077db56620986" } modify_external_catalogs: true
I20251023 20:15:15.048969 23145 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_45640565c4a24ff88274bc4e62b865ff" } modify_external_catalogs: true
I20251023 20:15:15.049013 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f66ae376b73c46d180bde83fb80dc7bb" } modify_external_catalogs: true
I20251023 20:15:15.049154 23145 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_45640565c4a24ff88274bc4e62b865ff" } modify_external_catalogs: true
I20251023 20:15:15.049276 23786 tablet_bootstrap.cc:654] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:15.052062 23584 tablet.cc:2378] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:15.056968 23787 tablet_bootstrap.cc:492] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:15.057155 23716 heartbeater.cc:646] Failed to heartbeat to 127.22.127.190:34331 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:15.063323 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3035573f53c4a6db0bb72882ca84f37 consensus state has no leader
I20251023 20:15:15.063632 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 799132aedc784de3bf0fc6478c8ef496 (DEFAULT_TABLE table=default.Table_a90cbb8972924d6aad10abd0327c9874 [id=b2d8f2d64c7e4668a8e274bcdc384754]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.063702 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_579a7389d3174efdbaf6d1724548bf04" } modify_external_catalogs: true
I20251023 20:15:15.063886 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_579a7389d3174efdbaf6d1724548bf04" } modify_external_catalogs: true
I20251023 20:15:15.064113 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799132aedc784de3bf0fc6478c8ef496. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.064301 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e1cce6823663453ba599faec46826631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.064831 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1cce6823663453ba599faec46826631 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.069710 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 799132aedc784de3bf0fc6478c8ef496 (DEFAULT_TABLE table=default.Table_a90cbb8972924d6aad10abd0327c9874 [id=b2d8f2d64c7e4668a8e274bcdc384754]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.069911 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.070154 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799132aedc784de3bf0fc6478c8ef496. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.071076 23784 tablet_replica.cc:333] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.071584 23784 raft_consensus.cc:2243] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.072026 23784 raft_consensus.cc:2272] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.072542 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d9b25166b00842948980cdfcacdd5df2
I20251023 20:15:15.073256 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8722fa0b5b6e4386af6870ce0b1f7963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.073781 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8722fa0b5b6e4386af6870ce0b1f7963 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.073922 23784 ts_tablet_manager.cc:1916] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.074584 23787 ts_tablet_manager.cc:1403] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.233s user 0.085s sys 0.091s
I20251023 20:15:15.075796 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_2bcd3241a40b49e2894f0104cebaad30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.076280 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bcd3241a40b49e2894f0104cebaad30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.074709 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 799132aedc784de3bf0fc6478c8ef496 (DEFAULT_TABLE table=default.Table_a90cbb8972924d6aad10abd0327c9874 [id=b2d8f2d64c7e4668a8e274bcdc384754]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.078374 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799132aedc784de3bf0fc6478c8ef496. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.078465 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet d9b25166b00842948980cdfcacdd5df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:15.079107 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d9b25166b00842948980cdfcacdd5df2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9b25166b00842948980cdfcacdd5df2 already in progress: creating tablet
I20251023 20:15:15.079501 23790 log.cc:826] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:15.083765 23784 ts_tablet_manager.cc:1929] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.083992 23784 log.cc:1199] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/2631175754104e4a9821849f27f04ecb
I20251023 20:15:15.084510 23784 ts_tablet_manager.cc:1950] T 2631175754104e4a9821849f27f04ecb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.086099 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_39878ca4314e43b8bf37b4cba1a5fb54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.086736 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39878ca4314e43b8bf37b4cba1a5fb54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.088307 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 2631175754104e4a9821849f27f04ecb (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:15.088354 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet d9b25166b00842948980cdfcacdd5df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.088454 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet d9b25166b00842948980cdfcacdd5df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.090195 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9b25166b00842948980cdfcacdd5df2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9b25166b00842948980cdfcacdd5df2 already in progress: creating tablet
W20251023 20:15:15.092061 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9b25166b00842948980cdfcacdd5df2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9b25166b00842948980cdfcacdd5df2 already in progress: creating tablet
I20251023 20:15:15.093068 23787 raft_consensus.cc:359] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.097674 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 24528de9d0a84eddb091dec4f1d2b4e2) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.098035 23784 tablet_replica.cc:333] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.100564 23364 tablet_service.cc:1505] Processing CreateTablet for tablet 955412de8bd1482ea7304872b48f38c0 (DEFAULT_TABLE table=default.Table_212585028f354c3e9403737b7ea1c732 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.101159 23364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955412de8bd1482ea7304872b48f38c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.103703 23361 tablet_service.cc:1505] Processing CreateTablet for tablet a2273d5581784756b737f8c7c17ed3a9 (DEFAULT_TABLE table=default.Table_156963dcf5454475a0cae4d52eff17f2 [id=9ed392ddcba74cd4a629eb0b728171ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.104089 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2273d5581784756b737f8c7c17ed3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.105468 23369 tablet_service.cc:1505] Processing CreateTablet for tablet dc8f615c621043da88d52f3964b0a0ec (DEFAULT_TABLE table=default.Table_344bdb8a8edd4641b4e9786c5d6ce520 [id=d7983b16809245a2a73b2a6cd79f6e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.105304 23650 tablet_service.cc:1505] Processing CreateTablet for tablet a2273d5581784756b737f8c7c17ed3a9 (DEFAULT_TABLE table=default.Table_156963dcf5454475a0cae4d52eff17f2 [id=9ed392ddcba74cd4a629eb0b728171ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.105830 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc8f615c621043da88d52f3964b0a0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.105847 23504 tablet_service.cc:1505] Processing CreateTablet for tablet a2273d5581784756b737f8c7c17ed3a9 (DEFAULT_TABLE table=default.Table_156963dcf5454475a0cae4d52eff17f2 [id=9ed392ddcba74cd4a629eb0b728171ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.106036 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2273d5581784756b737f8c7c17ed3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.106202 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2273d5581784756b737f8c7c17ed3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.105630 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 955412de8bd1482ea7304872b48f38c0 (DEFAULT_TABLE table=default.Table_212585028f354c3e9403737b7ea1c732 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.109448 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955412de8bd1482ea7304872b48f38c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.109501 23363 tablet_service.cc:1505] Processing CreateTablet for tablet bd88285bf0a94fad858a25004609131c (DEFAULT_TABLE table=default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.109843 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd88285bf0a94fad858a25004609131c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.110299 23645 tablet_service.cc:1505] Processing CreateTablet for tablet dc8f615c621043da88d52f3964b0a0ec (DEFAULT_TABLE table=default.Table_344bdb8a8edd4641b4e9786c5d6ce520 [id=d7983b16809245a2a73b2a6cd79f6e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.110531 23642 tablet_service.cc:1505] Processing CreateTablet for tablet cc5ce157b6ca42da90b051a184983bfe (DEFAULT_TABLE table=default.Table_bd046e3e4fd9439c8aabbcee605207e4 [id=8b2cfa043c8147098454bfead6809936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.110641 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc8f615c621043da88d52f3964b0a0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.110903 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5ce157b6ca42da90b051a184983bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.111172 23647 tablet_service.cc:1505] Processing CreateTablet for tablet bd88285bf0a94fad858a25004609131c (DEFAULT_TABLE table=default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.111518 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd88285bf0a94fad858a25004609131c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.113547 23365 tablet_service.cc:1505] Processing CreateTablet for tablet cc5ce157b6ca42da90b051a184983bfe (DEFAULT_TABLE table=default.Table_bd046e3e4fd9439c8aabbcee605207e4 [id=8b2cfa043c8147098454bfead6809936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.113883 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5ce157b6ca42da90b051a184983bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.113878 23505 tablet_service.cc:1505] Processing CreateTablet for tablet bd88285bf0a94fad858a25004609131c (DEFAULT_TABLE table=default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.114238 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd88285bf0a94fad858a25004609131c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.115759 23787 raft_consensus.cc:385] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.115871 23787 raft_consensus.cc:740] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.116125 23515 tablet_service.cc:1505] Processing CreateTablet for tablet dc8f615c621043da88d52f3964b0a0ec (DEFAULT_TABLE table=default.Table_344bdb8a8edd4641b4e9786c5d6ce520 [id=d7983b16809245a2a73b2a6cd79f6e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.116457 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc8f615c621043da88d52f3964b0a0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.116660 23787 consensus_queue.cc:260] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.117379 23501 tablet_service.cc:1505] Processing CreateTablet for tablet cc5ce157b6ca42da90b051a184983bfe (DEFAULT_TABLE table=default.Table_bd046e3e4fd9439c8aabbcee605207e4 [id=8b2cfa043c8147098454bfead6809936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.117749 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5ce157b6ca42da90b051a184983bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.132565 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 222ee7b9fa0a4bf69bb2ce8dab366f19
I20251023 20:15:15.133117 23139 catalog_manager.cc:6719] ReplaceTablet: tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 deleted and replaced by tablet 5416da30c66645c19d36112980608527
I20251023 20:15:15.133636 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 216e2c887a364c8ea5068c3834d1e4e2
I20251023 20:15:15.136762 23790 tablet_bootstrap.cc:492] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.137029 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:15.138193 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 216e2c887a364c8ea5068c3834d1e4e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 216e2c887a364c8ea5068c3834d1e4e2 already in progress: creating tablet
I20251023 20:15:15.138270 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.137285 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 with delete_type TABLET_DATA_DELETED (replaced by tablet 5416da30c66645c19d36112980608527) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:15.138993 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 already in progress: creating tablet
I20251023 20:15:15.139220 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 with delete_type TABLET_DATA_DELETED (replaced by tablet 5416da30c66645c19d36112980608527) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.139541 23790 ts_tablet_manager.cc:1403] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.132s user 0.000s sys 0.010s
I20251023 20:15:15.139554 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.140228 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 216e2c887a364c8ea5068c3834d1e4e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 216e2c887a364c8ea5068c3834d1e4e2 already in progress: creating tablet
I20251023 20:15:15.140640 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 with delete_type TABLET_DATA_DELETED (replaced by tablet 5416da30c66645c19d36112980608527) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.141268 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 already in progress: creating tablet
I20251023 20:15:15.152593 23786 tablet_bootstrap.cc:492] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.152719 23787 ts_tablet_manager.cc:1434] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.078s user 0.022s sys 0.017s
I20251023 20:15:15.152767 23786 ts_tablet_manager.cc:1403] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.105s user 0.000s sys 0.025s
I20251023 20:15:15.153688 23786 raft_consensus.cc:359] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.153980 23786 raft_consensus.cc:385] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.154120 23786 raft_consensus.cc:740] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.154426 23786 consensus_queue.cc:260] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.156486 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17
I20251023 20:15:15.158113 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:15.158834 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 already in progress: creating tablet
I20251023 20:15:15.159201 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.159629 23794 raft_consensus.cc:493] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:15.159844 23794 raft_consensus.cc:515] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.161896 23787 tablet_bootstrap.cc:492] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.161866 23794 leader_election.cc:290] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:15.163820 23787 tablet_bootstrap.cc:654] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.166090 23786 ts_tablet_manager.cc:1434] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251023 20:15:15.168641 23786 tablet_bootstrap.cc:492] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.169052 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet b09c0412f1eb48df8de542ca687a7b48
I20251023 20:15:15.170053 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet c3035573f53c4a6db0bb72882ca84f37
I20251023 20:15:15.170480 23140 catalog_manager.cc:6719] ReplaceTablet: tablet c3035573f53c4a6db0bb72882ca84f37 deleted and replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc
I20251023 20:15:15.171123 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet bd41c9c84d814f9191d5952e71e31a06
I20251023 20:15:15.171417 23786 tablet_bootstrap.cc:654] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.171510 23141 catalog_manager.cc:6719] ReplaceTablet: tablet bd41c9c84d814f9191d5952e71e31a06 deleted and replaced by tablet 34a738510f514b088469de3d8a39ca2c
I20251023 20:15:15.172034 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet b125422cf17145d5aef418817082eb32
I20251023 20:15:15.172418 23144 catalog_manager.cc:6719] ReplaceTablet: tablet b125422cf17145d5aef418817082eb32 deleted and replaced by tablet e44dfb4fd3d04a58b5fcc6102e055ff1
I20251023 20:15:15.176393 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 091e54a94f2d4c40a0006c7e35f74823
I20251023 20:15:15.176195 23790 raft_consensus.cc:359] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.176632 23790 raft_consensus.cc:385] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.176733 23790 raft_consensus.cc:740] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.179781 23787 tablet_bootstrap.cc:492] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.179953 23787 ts_tablet_manager.cc:1403] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20251023 20:15:15.180850 23787 raft_consensus.cc:359] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.181056 23787 raft_consensus.cc:385] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.181181 23787 raft_consensus.cc:740] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.181406 23787 consensus_queue.cc:260] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.182021 23787 ts_tablet_manager.cc:1434] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:15.182317 23787 tablet_bootstrap.cc:492] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.183063 23790 consensus_queue.cc:260] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.183771 23787 tablet_bootstrap.cc:654] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.185374 23786 tablet_bootstrap.cc:492] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.186365 23786 ts_tablet_manager.cc:1403] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20251023 20:15:15.187495 23786 raft_consensus.cc:359] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.189344 23786 raft_consensus.cc:385] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.189728 23786 raft_consensus.cc:740] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.191782 23790 ts_tablet_manager.cc:1434] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.052s user 0.009s sys 0.002s
I20251023 20:15:15.192140 23786 consensus_queue.cc:260] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.098651 23784 raft_consensus.cc:2243] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.194165 23784 raft_consensus.cc:2272] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.194928 23784 ts_tablet_manager.cc:1916] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.201332 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.202100 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet b09c0412f1eb48df8de542ca687a7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.202658 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet c3035573f53c4a6db0bb72882ca84f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.203208 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet bd41c9c84d814f9191d5952e71e31a06 with delete_type TABLET_DATA_DELETED (replaced by tablet 34a738510f514b088469de3d8a39ca2c) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.203490 23783 tablet_replica.cc:333] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.203998 23783 raft_consensus.cc:2243] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.204396 23783 raft_consensus.cc:2272] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.204602 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 955412de8bd1482ea7304872b48f38c0 (DEFAULT_TABLE table=default.Table_212585028f354c3e9403737b7ea1c732 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.204983 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955412de8bd1482ea7304872b48f38c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.206169 23783 ts_tablet_manager.cc:1916] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.209434 23786 ts_tablet_manager.cc:1434] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20251023 20:15:15.213063 23783 ts_tablet_manager.cc:1929] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:15.216372 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 already in progress: creating tablet
W20251023 20:15:15.217170 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet b09c0412f1eb48df8de542ca687a7b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b09c0412f1eb48df8de542ca687a7b48 already in progress: creating tablet
W20251023 20:15:15.217579 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c3035573f53c4a6db0bb72882ca84f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3035573f53c4a6db0bb72882ca84f37 already in progress: creating tablet
I20251023 20:15:15.213601 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet b125422cf17145d5aef418817082eb32 with delete_type TABLET_DATA_DELETED (replaced by tablet e44dfb4fd3d04a58b5fcc6102e055ff1) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.213886 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.214341 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 (DEFAULT_TABLE table=default.Table_2bcd3241a40b49e2894f0104cebaad30 [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.226588 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.228174 23784 ts_tablet_manager.cc:1929] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.228320 23784 log.cc:1199] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/9a2d1315afa0468c956adf64e1bd9cf0
I20251023 20:15:15.228693 23784 ts_tablet_manager.cc:1950] T 9a2d1315afa0468c956adf64e1bd9cf0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.214854 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 949e31d4a9ee4579843ed54153e22c7d (DEFAULT_TABLE table=default.Table_8722fa0b5b6e4386af6870ce0b1f7963 [id=a7821323b5954830ad19f8f771bfe6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.215329 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 (DEFAULT_TABLE table=default.Table_39878ca4314e43b8bf37b4cba1a5fb54 [id=c1b3327a41c8461cb83d9b31afa03e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.229378 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 949e31d4a9ee4579843ed54153e22c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.229624 23783 log.cc:1199] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/bd41c9c84d814f9191d5952e71e31a06
I20251023 20:15:15.229986 23787 tablet_bootstrap.cc:492] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.230170 23787 ts_tablet_manager.cc:1403] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.013s
I20251023 20:15:15.230269 23783 ts_tablet_manager.cc:1950] T bd41c9c84d814f9191d5952e71e31a06 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.230942 23787 raft_consensus.cc:359] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.231158 23787 raft_consensus.cc:385] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.231266 23787 raft_consensus.cc:740] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.231513 23787 consensus_queue.cc:260] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.232031 23787 ts_tablet_manager.cc:1434] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:15.215730 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.215816 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 1685213b588148febd6a23e1d2d6691d (DEFAULT_TABLE table=default.Table_e1cce6823663453ba599faec46826631 [id=580a286a829e44f9b9992f3b0b7482eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.218323 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 34a738510f514b088469de3d8a39ca2c (DEFAULT_TABLE table=default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.220747 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc (DEFAULT_TABLE table=default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.225497 23644 tablet_service.cc:1505] Processing CreateTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 (DEFAULT_TABLE table=default.Table_e81dbc08f93d42d18c38eccb996633d4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.225951 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 5416da30c66645c19d36112980608527 (DEFAULT_TABLE table=default.Table_877570fc930d464e8348f118f840439c [id=6e38fec2eebd48b8a6364be16ec99997]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.233712 23786 tablet_bootstrap.cc:492] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.235404 23786 tablet_bootstrap.cc:654] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.237010 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 9a2d1315afa0468c956adf64e1bd9cf0 (table default.Table_46ae5f307de94522a026332b6aec3650 [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
W20251023 20:15:15.237428 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 216e2c887a364c8ea5068c3834d1e4e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 216e2c887a364c8ea5068c3834d1e4e2 already in progress: creating tablet
W20251023 20:15:15.237815 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 already in progress: creating tablet
W20251023 20:15:15.238175 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 already in progress: creating tablet
I20251023 20:15:15.239051 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet c3035573f53c4a6db0bb72882ca84f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.239382 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet bd41c9c84d814f9191d5952e71e31a06 with delete_type TABLET_DATA_DELETED (replaced by tablet 34a738510f514b088469de3d8a39ca2c) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.239735 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet b125422cf17145d5aef418817082eb32 with delete_type TABLET_DATA_DELETED (replaced by tablet e44dfb4fd3d04a58b5fcc6102e055ff1) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.240041 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.223891 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet b09c0412f1eb48df8de542ca687a7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.224359 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet c3035573f53c4a6db0bb72882ca84f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.240597 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 (DEFAULT_TABLE table=default.Table_2bcd3241a40b49e2894f0104cebaad30 [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.240970 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.224525 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet bd41c9c84d814f9191d5952e71e31a06 with delete_type TABLET_DATA_DELETED (replaced by tablet 34a738510f514b088469de3d8a39ca2c) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.224705 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet b125422cf17145d5aef418817082eb32 with delete_type TABLET_DATA_DELETED (replaced by tablet e44dfb4fd3d04a58b5fcc6102e055ff1) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:15.241571 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet b09c0412f1eb48df8de542ca687a7b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b09c0412f1eb48df8de542ca687a7b48 already in progress: creating tablet
W20251023 20:15:15.241827 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c3035573f53c4a6db0bb72882ca84f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3035573f53c4a6db0bb72882ca84f37 already in progress: creating tablet
I20251023 20:15:15.224875 23363 tablet_service.cc:1552] Processing DeleteTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.225229 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 (DEFAULT_TABLE table=default.Table_39878ca4314e43b8bf37b4cba1a5fb54 [id=c1b3327a41c8461cb83d9b31afa03e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.242319 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c75cce4b9d4c48800b2d0908676a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.225426 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 949e31d4a9ee4579843ed54153e22c7d (DEFAULT_TABLE table=default.Table_8722fa0b5b6e4386af6870ce0b1f7963 [id=a7821323b5954830ad19f8f771bfe6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.245182 23783 tablet_replica.cc:333] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.245358 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 949e31d4a9ee4579843ed54153e22c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.245371 23783 raft_consensus.cc:2243] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.245507 23783 raft_consensus.cc:2272] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.245872 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1685213b588148febd6a23e1d2d6691d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.225571 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 (DEFAULT_TABLE table=default.Table_2bcd3241a40b49e2894f0104cebaad30 [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.225850 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 1685213b588148febd6a23e1d2d6691d (DEFAULT_TABLE table=default.Table_e1cce6823663453ba599faec46826631 [id=580a286a829e44f9b9992f3b0b7482eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.249235 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1685213b588148febd6a23e1d2d6691d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.250023 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.250022 23783 ts_tablet_manager.cc:1916] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.251632 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a738510f514b088469de3d8a39ca2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.225921 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 34a738510f514b088469de3d8a39ca2c (DEFAULT_TABLE table=default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.255981 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a738510f514b088469de3d8a39ca2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.256042 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet bd41c9c84d814f9191d5952e71e31a06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd41c9c84d814f9191d5952e71e31a06 already in progress: creating tablet
W20251023 20:15:15.256311 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet b125422cf17145d5aef418817082eb32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b125422cf17145d5aef418817082eb32 already in progress: creating tablet
W20251023 20:15:15.256508 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 091e54a94f2d4c40a0006c7e35f74823 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 091e54a94f2d4c40a0006c7e35f74823 already in progress: creating tablet
I20251023 20:15:15.256808 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b4abd308054bac9d49b1d8e7dd2ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.226402 23379 tablet_service.cc:1505] Processing CreateTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc (DEFAULT_TABLE table=default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.259877 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b4abd308054bac9d49b1d8e7dd2ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.261330 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44dfb4fd3d04a58b5fcc6102e055ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.261955 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 949e31d4a9ee4579843ed54153e22c7d (DEFAULT_TABLE table=default.Table_8722fa0b5b6e4386af6870ce0b1f7963 [id=a7821323b5954830ad19f8f771bfe6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.262367 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 949e31d4a9ee4579843ed54153e22c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.226886 23367 tablet_service.cc:1505] Processing CreateTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 (DEFAULT_TABLE table=default.Table_e81dbc08f93d42d18c38eccb996633d4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.264819 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44dfb4fd3d04a58b5fcc6102e055ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.265309 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5416da30c66645c19d36112980608527. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.227340 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 5416da30c66645c19d36112980608527 (DEFAULT_TABLE table=default.Table_877570fc930d464e8348f118f840439c [id=6e38fec2eebd48b8a6364be16ec99997]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.265600 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5416da30c66645c19d36112980608527. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.265671 23783 ts_tablet_manager.cc:1929] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.265800 23783 log.cc:1199] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/2631175754104e4a9821849f27f04ecb
I20251023 20:15:15.266144 23783 ts_tablet_manager.cc:1950] T 2631175754104e4a9821849f27f04ecb P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.268656 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 (DEFAULT_TABLE table=default.Table_39878ca4314e43b8bf37b4cba1a5fb54 [id=c1b3327a41c8461cb83d9b31afa03e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.269027 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c75cce4b9d4c48800b2d0908676a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.269277 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c75cce4b9d4c48800b2d0908676a14. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.269371 23718 tablet.cc:2378] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:15.227696 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.275821 23790 tablet_bootstrap.cc:492] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.277679 23790 tablet_bootstrap.cc:654] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.277838 23785 tablet_replica.cc:333] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.278357 23785 raft_consensus.cc:2243] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.278522 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.278792 23784 tablet_replica.cc:333] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.278787 23785 raft_consensus.cc:2272] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.278937 23784 raft_consensus.cc:2243] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.279089 23784 raft_consensus.cc:2272] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.279513 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet b09c0412f1eb48df8de542ca687a7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.279706 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.280042 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet dd456675077a4c73933c26e9a0313848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.280372 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 2177416169ca4d0d8368b1cac90c513e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.281062 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 34a738510f514b088469de3d8a39ca2c (DEFAULT_TABLE table=default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.281436 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a738510f514b088469de3d8a39ca2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.284680 23787 tablet_bootstrap.cc:492] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.285581 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_9dd11819a7d24778856f462839d28a31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.286124 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dd11819a7d24778856f462839d28a31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.286410 23787 tablet_bootstrap.cc:654] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.287300 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ea61082402e14334acc5162a368129e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.287411 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc (DEFAULT_TABLE table=default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.287732 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73b4abd308054bac9d49b1d8e7dd2ebc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.287755 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea61082402e14334acc5162a368129e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.288331 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet bd41c9c84d814f9191d5952e71e31a06 (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]) successfully deleted
W20251023 20:15:15.288758 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet b125422cf17145d5aef418817082eb32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b125422cf17145d5aef418817082eb32 already in progress: creating tablet
I20251023 20:15:15.288868 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_853f1904ea4d44919b14dc452f631d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.289129 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 091e54a94f2d4c40a0006c7e35f74823 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 091e54a94f2d4c40a0006c7e35f74823 already in progress: creating tablet
I20251023 20:15:15.289299 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_877570fc930d464e8348f118f840439c" } new_table_name: "default.Table_f01b7fbaa8924dfa965e4f6d3529dd2e" modify_external_catalogs: true
I20251023 20:15:15.289575 23784 ts_tablet_manager.cc:1916] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.289674 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 2631175754104e4a9821849f27f04ecb (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:15.290387 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b2975466aafc47948a16bf139c0d70b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.290855 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2975466aafc47948a16bf139c0d70b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.291850 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a6d0f3fe1c734fac92db5f6868a805de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.292253 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6d0f3fe1c734fac92db5f6868a805de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.293239 23145 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c9b696fd225e430195d01e0efc28f219" } new_table_name: "default.Table_24d0ef155c144a43828c4de4e53e0c00" modify_external_catalogs: true
I20251023 20:15:15.293371 23507 tablet_service.cc:1505] Processing CreateTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 (DEFAULT_TABLE table=default.Table_e81dbc08f93d42d18c38eccb996633d4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.293784 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44dfb4fd3d04a58b5fcc6102e055ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.294286 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d668e9c8e1394c4486b749d97aa78127"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.294694 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d668e9c8e1394c4486b749d97aa78127 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.295658 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_46ae5f307de94522a026332b6aec3650" } new_table_name: "default.Table_6a1c8c9085ab4216b3dbfaa3a23ef61d" modify_external_catalogs: true
I20251023 20:15:15.296666 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_212585028f354c3e9403737b7ea1c732" } new_table_name: "default.Table_837a7dbe47ef4dd3ba76317c430a26b1" modify_external_catalogs: true
I20251023 20:15:15.296882 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 5416da30c66645c19d36112980608527 (DEFAULT_TABLE table=default.Table_877570fc930d464e8348f118f840439c [id=6e38fec2eebd48b8a6364be16ec99997]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.297224 23784 ts_tablet_manager.cc:1929] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.297255 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5416da30c66645c19d36112980608527. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.297839 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c3035573f53c4a6db0bb72882ca84f37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3035573f53c4a6db0bb72882ca84f37 already in progress: creating tablet
W20251023 20:15:15.298215 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet bd41c9c84d814f9191d5952e71e31a06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd41c9c84d814f9191d5952e71e31a06 already in progress: creating tablet
W20251023 20:15:15.298537 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet b125422cf17145d5aef418817082eb32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b125422cf17145d5aef418817082eb32 already in progress: creating tablet
W20251023 20:15:15.298717 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_853f1904ea4d44919b14dc452f631d2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:15.298916 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 091e54a94f2d4c40a0006c7e35f74823 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 091e54a94f2d4c40a0006c7e35f74823 already in progress: creating tablet
I20251023 20:15:15.302266 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd456675077a4c73933c26e9a0313848" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:15.303779 23784 log.cc:1199] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/6ae1876fa334495ca95dc2e9e6b4a0fd
I20251023 20:15:15.304209 23784 ts_tablet_manager.cc:1950] T 6ae1876fa334495ca95dc2e9e6b4a0fd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.305327 23784 tablet_replica.cc:333] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.305518 23784 raft_consensus.cc:2243] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.305677 23784 raft_consensus.cc:2272] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.306277 23785 ts_tablet_manager.cc:1916] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.306411 23784 ts_tablet_manager.cc:1916] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.309494 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet e87648998c0b4297ab644d2c39ab2132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.309820 23783 tablet_replica.cc:333] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.310930 23783 raft_consensus.cc:2243] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.311139 23783 raft_consensus.cc:2272] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.310674 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd456675077a4c73933c26e9a0313848" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:15.312660 23783 ts_tablet_manager.cc:1916] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:15.312960 23470 leader_election.cc:343] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:15.317587 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 6ae1876fa334495ca95dc2e9e6b4a0fd (table default.Table_8b24573ba4864a13a123e5d8d2bf9448 [id=fb7af3cee68a4f3c9307a5876db279c1]) successfully deleted
W20251023 20:15:15.318094 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet b09c0412f1eb48df8de542ca687a7b48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b09c0412f1eb48df8de542ca687a7b48 already in progress: creating tablet
W20251023 20:15:15.318476 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2631175754104e4a9821849f27f04ecb because the tablet was not found. No further retry: Not found: Tablet not found: 2631175754104e4a9821849f27f04ecb
W20251023 20:15:15.318828 23471 leader_election.cc:343] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:15.319021 23471 leader_election.cc:304] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
W20251023 20:15:15.319438 23471 raft_consensus.cc:2686] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251023 20:15:15.319533 23783 ts_tablet_manager.cc:1929] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.319677 23783 log.cc:1199] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e87648998c0b4297ab644d2c39ab2132
I20251023 20:15:15.320016 23783 ts_tablet_manager.cc:1950] T e87648998c0b4297ab644d2c39ab2132 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.321012 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet e87648998c0b4297ab644d2c39ab2132 (table default.Table_57c800e050ac4e75a5cdbfff336ec344 [id=823681dedfc3477d995ad3695c4a9541]) successfully deleted
I20251023 20:15:15.321916 23786 tablet_bootstrap.cc:492] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.322110 23786 ts_tablet_manager.cc:1403] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.001s
I20251023 20:15:15.322926 23786 raft_consensus.cc:359] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.323225 23786 raft_consensus.cc:385] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.323367 23786 raft_consensus.cc:740] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.323662 23786 consensus_queue.cc:260] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.325862 23786 ts_tablet_manager.cc:1434] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:15.326174 23786 tablet_bootstrap.cc:492] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.327811 23786 tablet_bootstrap.cc:654] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.334900 23790 tablet_bootstrap.cc:492] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.335121 23790 ts_tablet_manager.cc:1403] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20251023 20:15:15.336176 23790 raft_consensus.cc:359] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.336436 23790 raft_consensus.cc:385] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.336570 23790 raft_consensus.cc:740] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.336891 23790 consensus_queue.cc:260] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.339406 23790 ts_tablet_manager.cc:1434] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:15.340466 23790 tablet_bootstrap.cc:492] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.336869 23784 ts_tablet_manager.cc:1929] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.345237 23784 log.cc:1199] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/dd456675077a4c73933c26e9a0313848
I20251023 20:15:15.345690 23784 ts_tablet_manager.cc:1950] T dd456675077a4c73933c26e9a0313848 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.346647 23784 tablet_replica.cc:333] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.346817 23784 raft_consensus.cc:2243] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.346980 23784 raft_consensus.cc:2272] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.347455 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet dd456675077a4c73933c26e9a0313848 (table default.Table_da0691e8e6224cfebf17fe7df75954dd [id=a9e1acb22d6f423f924ad7417f9a986e]) successfully deleted
I20251023 20:15:15.347709 23784 ts_tablet_manager.cc:1916] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.348132 23790 tablet_bootstrap.cc:654] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.354664 23784 ts_tablet_manager.cc:1929] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:15.359170 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24528de9d0a84eddb091dec4f1d2b4e2 consensus state has no leader
I20251023 20:15:15.360639 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e1cce6823663453ba599faec46826631" } new_table_name: "default.Table_b199143b364a43798ec81927ae2148ca" modify_external_catalogs: true
I20251023 20:15:15.361318 23784 log.cc:1199] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/2177416169ca4d0d8368b1cac90c513e
I20251023 20:15:15.361694 23790 tablet_bootstrap.cc:492] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.361716 23784 ts_tablet_manager.cc:1950] T 2177416169ca4d0d8368b1cac90c513e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.361866 23790 ts_tablet_manager.cc:1403] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20251023 20:15:15.360037 23786 tablet_bootstrap.cc:492] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
W20251023 20:15:15.362111 23145 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0232c33db59641b3af27bfd15cd7fb81 consensus state has no leader
I20251023 20:15:15.362124 23786 ts_tablet_manager.cc:1403] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.003s
I20251023 20:15:15.362476 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_bf1bef05ff6749359d1c5be2beb1da6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.362805 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 2177416169ca4d0d8368b1cac90c513e (table default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3 [id=2d40a1f71ab847df9127c80fd25f05af]) successfully deleted
W20251023 20:15:15.362983 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf1bef05ff6749359d1c5be2beb1da6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.363646 23786 raft_consensus.cc:359] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.363902 23786 raft_consensus.cc:385] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.364017 23786 raft_consensus.cc:740] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:15.364141 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5416da30c66645c19d36112980608527 consensus state has no leader
I20251023 20:15:15.364352 23786 consensus_queue.cc:260] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.364449 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_57a2a81ca5484113b5e2a0857d40dbbb" } modify_external_catalogs: true
I20251023 20:15:15.364622 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_57a2a81ca5484113b5e2a0857d40dbbb" } modify_external_catalogs: true
I20251023 20:15:15.365340 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet b125422cf17145d5aef418817082eb32 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.365876 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_baf231bb8eaa40eeb01169ddcf5161bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.366357 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baf231bb8eaa40eeb01169ddcf5161bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.367769 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_3320634019ac4bf48649eff33160d808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.368189 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3320634019ac4bf48649eff33160d808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.368798 23786 ts_tablet_manager.cc:1434] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:15.369660 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_561d6afe24b246c9909547a248a5fe7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.370077 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_561d6afe24b246c9909547a248a5fe7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.371977 23786 tablet_bootstrap.cc:492] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.372144 23790 raft_consensus.cc:359] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.374418 23790 raft_consensus.cc:385] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.374572 23790 raft_consensus.cc:740] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.374873 23790 consensus_queue.cc:260] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.374081 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.376886 23790 ts_tablet_manager.cc:1434] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251023 20:15:15.379103 23790 tablet_bootstrap.cc:492] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.379722 23786 tablet_bootstrap.cc:654] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.381474 23790 tablet_bootstrap.cc:654] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.384217 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet b125422cf17145d5aef418817082eb32 with delete_type TABLET_DATA_DELETED (replaced by tablet e44dfb4fd3d04a58b5fcc6102e055ff1) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.384524 23784 tablet_replica.cc:333] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.384752 23784 raft_consensus.cc:2243] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.384927 23784 raft_consensus.cc:2272] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.385840 23784 ts_tablet_manager.cc:1916] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.392311 23784 ts_tablet_manager.cc:1929] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.392457 23784 log.cc:1199] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/b125422cf17145d5aef418817082eb32
I20251023 20:15:15.392773 23784 ts_tablet_manager.cc:1950] T b125422cf17145d5aef418817082eb32 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.393580 23785 ts_tablet_manager.cc:1929] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.393755 23785 log.cc:1199] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/fcec61131b7f4af590acf88a2ab3c5f1
I20251023 20:15:15.394186 23785 ts_tablet_manager.cc:1950] T fcec61131b7f4af590acf88a2ab3c5f1 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:15.394279 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet b125422cf17145d5aef418817082eb32 (table default.Table_e81dbc08f93d42d18c38eccb996633d4 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:15.395735 23785 tablet_replica.cc:333] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.395924 23785 raft_consensus.cc:2243] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.396087 23785 raft_consensus.cc:2272] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.396095 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet fcec61131b7f4af590acf88a2ab3c5f1 (table default.Table_07bcc06b3cdd42379508c3f649756c40 [id=0d5215e658b64dbcbc9d37fc2622ca61]) successfully deleted
I20251023 20:15:15.396556 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 1685213b588148febd6a23e1d2d6691d (DEFAULT_TABLE table=default.Table_e1cce6823663453ba599faec46826631 [id=580a286a829e44f9b9992f3b0b7482eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.396961 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1685213b588148febd6a23e1d2d6691d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.402196 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 955412de8bd1482ea7304872b48f38c0 consensus state has no leader
I20251023 20:15:15.403648 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bd046e3e4fd9439c8aabbcee605207e4" } modify_external_catalogs: true
I20251023 20:15:15.403824 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bd046e3e4fd9439c8aabbcee605207e4" } modify_external_catalogs: true
I20251023 20:15:15.406293 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 2631175754104e4a9821849f27f04ecb with delete_type TABLET_DATA_DELETED (replaced by tablet c3035573f53c4a6db0bb72882ca84f37) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.408779 23785 ts_tablet_manager.cc:1916] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.409848 23787 tablet_bootstrap.cc:492] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.410131 23787 ts_tablet_manager.cc:1403] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.126s user 0.049s sys 0.068s
I20251023 20:15:15.411311 23787 raft_consensus.cc:359] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.411623 23787 raft_consensus.cc:385] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.411748 23787 raft_consensus.cc:740] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.412034 23787 consensus_queue.cc:260] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.413336 23787 ts_tablet_manager.cc:1434] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:15.415324 23787 tablet_bootstrap.cc:492] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.415762 23785 ts_tablet_manager.cc:1929] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.417877 23786 tablet_bootstrap.cc:492] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.418083 23786 ts_tablet_manager.cc:1403] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.000s
I20251023 20:15:15.418896 23786 raft_consensus.cc:359] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.419103 23786 raft_consensus.cc:385] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.419198 23786 raft_consensus.cc:740] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.419435 23786 consensus_queue.cc:260] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.420567 23786 ts_tablet_manager.cc:1434] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:15.420881 23786 tablet_bootstrap.cc:492] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.417057 23787 tablet_bootstrap.cc:654] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.423389 23786 tablet_bootstrap.cc:654] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.416759 23790 tablet_bootstrap.cc:492] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.424340 23790 ts_tablet_manager.cc:1403] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20251023 20:15:15.425232 23790 raft_consensus.cc:359] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.425487 23790 raft_consensus.cc:385] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.425619 23790 raft_consensus.cc:740] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.425925 23790 consensus_queue.cc:260] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.428470 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet eedc0251bae548bfb4f1dd54f33b7a86 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.429550 23790 ts_tablet_manager.cc:1434] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:15.430312 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d592564ff9cd44ab86010457642460ec" } modify_external_catalogs: true
I20251023 20:15:15.430389 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 with delete_type TABLET_DATA_DELETED (replaced by tablet 5416da30c66645c19d36112980608527) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.431730 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d592564ff9cd44ab86010457642460ec" } modify_external_catalogs: true
I20251023 20:15:15.430981 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a90cbb8972924d6aad10abd0327c9874" } modify_external_catalogs: true
I20251023 20:15:15.432204 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a90cbb8972924d6aad10abd0327c9874" } modify_external_catalogs: true
I20251023 20:15:15.432296 23790 tablet_bootstrap.cc:492] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.432894 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet d9b25166b00842948980cdfcacdd5df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.433167 23784 tablet_replica.cc:333] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.433321 23784 raft_consensus.cc:2243] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.433482 23784 raft_consensus.cc:2272] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.433957 23790 tablet_bootstrap.cc:654] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.434868 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c (DEFAULT_TABLE table=default.Table_9dd11819a7d24778856f462839d28a31 [id=aea6c365745a4e50a1768a10d7803d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.435711 23784 ts_tablet_manager.cc:1916] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.437888 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c (DEFAULT_TABLE table=default.Table_9dd11819a7d24778856f462839d28a31 [id=aea6c365745a4e50a1768a10d7803d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.438287 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.441785 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c (DEFAULT_TABLE table=default.Table_9dd11819a7d24778856f462839d28a31 [id=aea6c365745a4e50a1768a10d7803d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.441952 23784 ts_tablet_manager.cc:1929] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.442049 23784 log.cc:1199] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d9b25166b00842948980cdfcacdd5df2
I20251023 20:15:15.442135 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.442389 23784 ts_tablet_manager.cc:1950] T d9b25166b00842948980cdfcacdd5df2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.443395 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.447079 23145 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2741dc8a0a2845cc945ac27e11d1bb98" } modify_external_catalogs: true
I20251023 20:15:15.447309 23145 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2741dc8a0a2845cc945ac27e11d1bb98" } modify_external_catalogs: true
W20251023 20:15:15.451507 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1685213b588148febd6a23e1d2d6691d consensus state has no leader
I20251023 20:15:15.451841 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet b09c0412f1eb48df8de542ca687a7b48 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.452450 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet b09c0412f1eb48df8de542ca687a7b48 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.453009 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet 091e54a94f2d4c40a0006c7e35f74823 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.452881 23785 log.cc:1199] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/091e54a94f2d4c40a0006c7e35f74823
I20251023 20:15:15.453572 23785 ts_tablet_manager.cc:1950] T 091e54a94f2d4c40a0006c7e35f74823 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:15.457057 23785 tablet_replica.cc:333] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.457116 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 091e54a94f2d4c40a0006c7e35f74823 (table default.Table_8e8a728158404934a7b077db56620986 [id=a7ccdb7723b44991be2809bf1d80d6a6]) successfully deleted
I20251023 20:15:15.457790 23785 raft_consensus.cc:2243] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.445459 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d9b25166b00842948980cdfcacdd5df2 (table default.Table_f66ae376b73c46d180bde83fb80dc7bb [id=4f40834f1fae488bb070b0c40c191ee8]) successfully deleted
I20251023 20:15:15.458760 23785 raft_consensus.cc:2272] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.460762 23785 ts_tablet_manager.cc:1916] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.465970 23786 tablet_bootstrap.cc:492] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.466187 23786 ts_tablet_manager.cc:1403] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.001s
I20251023 20:15:15.466987 23786 raft_consensus.cc:359] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:15.467247 23786 raft_consensus.cc:385] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.467360 23786 raft_consensus.cc:740] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.467635 23786 consensus_queue.cc:260] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:15.470077 23786 ts_tablet_manager.cc:1434] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:15.470927 23786 tablet_bootstrap.cc:492] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.472796 23786 tablet_bootstrap.cc:654] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.474283 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.474538 23783 tablet_replica.cc:333] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.474723 23783 raft_consensus.cc:2243] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.474884 23783 raft_consensus.cc:2272] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.475646 23783 ts_tablet_manager.cc:1916] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.475692 23787 tablet_bootstrap.cc:492] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.475850 23787 ts_tablet_manager.cc:1403] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.029s
I20251023 20:15:15.476356 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet d9ebd537f271426a95af432ec9070ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.476570 23784 tablet_replica.cc:333] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.476727 23784 raft_consensus.cc:2243] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.476686 23787 raft_consensus.cc:359] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.476867 23784 raft_consensus.cc:2272] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.476895 23787 raft_consensus.cc:385] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.476969 23787 raft_consensus.cc:740] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.477285 23787 consensus_queue.cc:260] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.478108 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.481758 23787 ts_tablet_manager.cc:1434] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251023 20:15:15.482671 23783 ts_tablet_manager.cc:1929] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.482869 23783 log.cc:1199] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/536216374bae4bc481d78fabbed3567f
I20251023 20:15:15.483289 23783 ts_tablet_manager.cc:1950] T 536216374bae4bc481d78fabbed3567f P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.484318 23787 tablet_bootstrap.cc:492] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.484732 23784 ts_tablet_manager.cc:1916] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.484725 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 536216374bae4bc481d78fabbed3567f (table default.Table_50a809e112244d0984bf222990b09883 [id=5991f18468ca47ac82800d008795e0dc]) successfully deleted
W20251023 20:15:15.485149 23132 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 1685213b588148febd6a23e1d2d6691d (table default.Table_b199143b364a43798ec81927ae2148ca [id=580a286a829e44f9b9992f3b0b7482eb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1685213b588148febd6a23e1d2d6691d consensus state has no leader
I20251023 20:15:15.486009 23787 tablet_bootstrap.cc:654] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.488862 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.489475 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 536216374bae4bc481d78fabbed3567f because the tablet was not found. No further retry: Not found: Tablet not found: 536216374bae4bc481d78fabbed3567f
I20251023 20:15:15.491640 23784 ts_tablet_manager.cc:1929] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.491992 23784 log.cc:1199] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d9ebd537f271426a95af432ec9070ae5
I20251023 20:15:15.492369 23784 ts_tablet_manager.cc:1950] T d9ebd537f271426a95af432ec9070ae5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.493346 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d9ebd537f271426a95af432ec9070ae5 (table default.Table_bbb88a1064eb4a27aa71756ff68bb6db [id=197fa361f58a4b24b5b394eb7f43b4e9]) successfully deleted
I20251023 20:15:15.493700 23784 tablet_replica.cc:333] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.493907 23784 raft_consensus.cc:2243] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.494146 23784 raft_consensus.cc:2272] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.494257 23787 tablet_bootstrap.cc:492] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.494405 23787 ts_tablet_manager.cc:1403] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20251023 20:15:15.494927 23784 ts_tablet_manager.cc:1916] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.495191 23787 raft_consensus.cc:359] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.495398 23787 raft_consensus.cc:385] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.495489 23787 raft_consensus.cc:740] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.495711 23787 consensus_queue.cc:260] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.497774 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.498082 23783 tablet_replica.cc:333] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.498247 23783 raft_consensus.cc:2243] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.498430 23783 raft_consensus.cc:2272] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.499063 23783 ts_tablet_manager.cc:1916] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.499444 23787 ts_tablet_manager.cc:1434] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:15.502352 23787 tablet_bootstrap.cc:492] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.504197 23787 tablet_bootstrap.cc:654] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.505733 23783 ts_tablet_manager.cc:1929] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.505978 23783 log.cc:1199] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/eedc0251bae548bfb4f1dd54f33b7a86
I20251023 20:15:15.506780 23142 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet d05145bbb50343f38166de6b35bad890 (table default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]). Expected version 1 got 0
I20251023 20:15:15.508554 23378 tablet_service.cc:1505] Processing CreateTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 (DEFAULT_TABLE table=default.Table_853f1904ea4d44919b14dc452f631d2f [id=b1abce94eb4c40b592048fca87a9e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.508846 23501 tablet_service.cc:1505] Processing CreateTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 (DEFAULT_TABLE table=default.Table_853f1904ea4d44919b14dc452f631d2f [id=b1abce94eb4c40b592048fca87a9e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.509009 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8aa8677f0ad4af6ae46b9ce731a7095. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.509418 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8aa8677f0ad4af6ae46b9ce731a7095. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.509413 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b (DEFAULT_TABLE table=default.Table_b2975466aafc47948a16bf139c0d70b1 [id=51c1f79e31c945309add87b2618ee778]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.509833 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b (DEFAULT_TABLE table=default.Table_b2975466aafc47948a16bf139c0d70b1 [id=51c1f79e31c945309add87b2618ee778]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.510181 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71263f7cf0994e78b5e7baa48fd4486b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.513197 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71263f7cf0994e78b5e7baa48fd4486b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.516878 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 (DEFAULT_TABLE table=default.Table_ea61082402e14334acc5162a368129e3 [id=1a48facca0024a5eaa670799650b9d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.517251 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d698ffe7d154ef8b1994ffdd4b0c648. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.517668 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 64f515399c494aadbed2bd0eac171a77 (DEFAULT_TABLE table=default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.518061 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f515399c494aadbed2bd0eac171a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.522794 23379 tablet_service.cc:1505] Processing CreateTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.523162 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f49a0e08de024e3b97cb2e6ac5a59c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.525326 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b (DEFAULT_TABLE table=default.Table_b2975466aafc47948a16bf139c0d70b1 [id=51c1f79e31c945309add87b2618ee778]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.526189 23142 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet d05145bbb50343f38166de6b35bad890 (table default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]). Expected version 1 got 0
I20251023 20:15:15.526616 23786 tablet_bootstrap.cc:492] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.526780 23786 ts_tablet_manager.cc:1403] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.002s
I20251023 20:15:15.527575 23786 raft_consensus.cc:359] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.527777 23786 raft_consensus.cc:385] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.527873 23786 raft_consensus.cc:740] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.528080 23786 consensus_queue.cc:260] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.528519 23637 tablet_service.cc:1505] Processing CreateTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.528983 23645 tablet_service.cc:1505] Processing CreateTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 (DEFAULT_TABLE table=default.Table_853f1904ea4d44919b14dc452f631d2f [id=b1abce94eb4c40b592048fca87a9e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.530378 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 0d73da547fd04557abd5aba73e76dd18
I20251023 20:15:15.531076 23790 tablet_bootstrap.cc:492] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.531246 23790 ts_tablet_manager.cc:1403] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.002s
I20251023 20:15:15.532018 23790 raft_consensus.cc:359] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.532210 23790 raft_consensus.cc:385] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.532280 23790 raft_consensus.cc:740] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.532833 23785 ts_tablet_manager.cc:1929] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.532945 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d73da547fd04557abd5aba73e76dd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.532999 23785 log.cc:1199] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/2631175754104e4a9821849f27f04ecb
I20251023 20:15:15.533138 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 799132aedc784de3bf0fc6478c8ef496
I20251023 20:15:15.533286 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 64f515399c494aadbed2bd0eac171a77 (DEFAULT_TABLE table=default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.533521 23787 tablet_bootstrap.cc:492] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.533780 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d8ad21e3888a4915ae5f252c2fca64cb
I20251023 20:15:15.534407 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet cc5ce157b6ca42da90b051a184983bfe
I20251023 20:15:15.534623 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d73da547fd04557abd5aba73e76dd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.534782 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 (DEFAULT_TABLE table=default.Table_ea61082402e14334acc5162a368129e3 [id=1a48facca0024a5eaa670799650b9d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.535444 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71263f7cf0994e78b5e7baa48fd4486b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.535706 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f515399c494aadbed2bd0eac171a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.536278 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f49a0e08de024e3b97cb2e6ac5a59c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.535466 23787 ts_tablet_manager.cc:1403] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.004s
I20251023 20:15:15.536785 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet cc5ce157b6ca42da90b051a184983bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.537420 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.537338 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_fb1a22feead44f33b0a8f5f5ec849493"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.537828 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d73da547fd04557abd5aba73e76dd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.537846 23786 ts_tablet_manager.cc:1434] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
W20251023 20:15:15.537907 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb1a22feead44f33b0a8f5f5ec849493 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.538120 23785 ts_tablet_manager.cc:1950] T 2631175754104e4a9821849f27f04ecb P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:15.538256 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 64f515399c494aadbed2bd0eac171a77 (DEFAULT_TABLE table=default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.538623 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f515399c494aadbed2bd0eac171a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.539011 23785 tablet_replica.cc:333] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.539182 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 2631175754104e4a9821849f27f04ecb (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:15.539230 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 73b4abd308054bac9d49b1d8e7dd2ebc
I20251023 20:15:15.539880 23786 tablet_bootstrap.cc:492] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.540517 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.540918 23785 raft_consensus.cc:2243] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.541415 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 799132aedc784de3bf0fc6478c8ef496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.541623 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.541653 23504 tablet_service.cc:1505] Processing CreateTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.542006 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f49a0e08de024e3b97cb2e6ac5a59c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.542308 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet cc5ce157b6ca42da90b051a184983bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.542347 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.542423 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet bd41c9c84d814f9191d5952e71e31a06 with delete_type TABLET_DATA_DELETED (replaced by tablet 34a738510f514b088469de3d8a39ca2c) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.541369 23785 raft_consensus.cc:2272] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.548961 23786 tablet_bootstrap.cc:654] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.549244 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 799132aedc784de3bf0fc6478c8ef496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.549811 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 (DEFAULT_TABLE table=default.Table_ea61082402e14334acc5162a368129e3 [id=1a48facca0024a5eaa670799650b9d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.541872 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 799132aedc784de3bf0fc6478c8ef496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.544422 23790 consensus_queue.cc:260] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.550603 23790 ts_tablet_manager.cc:1434] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20251023 20:15:15.550963 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d698ffe7d154ef8b1994ffdd4b0c648. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.551211 23790 tablet_bootstrap.cc:492] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.552742 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.190:34331
--webserver_interface=127.22.127.190
--webserver_port=44199
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251023 20:15:15.553918 23790 tablet_bootstrap.cc:654] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.556352 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8aa8677f0ad4af6ae46b9ce731a7095. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.559664 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet d9b25166b00842948980cdfcacdd5df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.560027 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.560222 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet cc5ce157b6ca42da90b051a184983bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.560374 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.560525 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d698ffe7d154ef8b1994ffdd4b0c648. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.561913 23787 raft_consensus.cc:359] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.565603 23785 ts_tablet_manager.cc:1916] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.566241 23784 ts_tablet_manager.cc:1929] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:15.568106 23447 tablet.cc:2378] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251023 20:15:15.569312 23783 ts_tablet_manager.cc:1950] T eedc0251bae548bfb4f1dd54f33b7a86 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.569412 23787 raft_consensus.cc:385] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.571969 23787 raft_consensus.cc:740] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.573468 23787 consensus_queue.cc:260] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.575114 23784 log.cc:1199] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/091e54a94f2d4c40a0006c7e35f74823
I20251023 20:15:15.575089 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet eedc0251bae548bfb4f1dd54f33b7a86 (table default.Table_c747a4f8ca7a461e90896f4145d9ceb6 [id=08f1b14fc24d49baa3869370048151aa]) successfully deleted
I20251023 20:15:15.575788 23784 ts_tablet_manager.cc:1950] T 091e54a94f2d4c40a0006c7e35f74823 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.578168 23787 ts_tablet_manager.cc:1434] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20251023 20:15:15.579330 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 091e54a94f2d4c40a0006c7e35f74823 (table default.Table_8e8a728158404934a7b077db56620986 [id=a7ccdb7723b44991be2809bf1d80d6a6]) successfully deleted
I20251023 20:15:15.579751 23787 tablet_bootstrap.cc:492] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:15.580166 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 0d73da547fd04557abd5aba73e76dd18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d73da547fd04557abd5aba73e76dd18 already in progress: creating tablet
I20251023 20:15:15.580404 23786 tablet_bootstrap.cc:492] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.580664 23786 ts_tablet_manager.cc:1403] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.001s
I20251023 20:15:15.581506 23786 raft_consensus.cc:359] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.581662 23787 tablet_bootstrap.cc:654] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.581795 23786 raft_consensus.cc:385] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.581923 23786 raft_consensus.cc:740] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.582252 23786 consensus_queue.cc:260] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.584095 23784 tablet_replica.cc:333] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.584285 23784 raft_consensus.cc:2243] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.584419 23784 raft_consensus.cc:2272] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:15.584522 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 0d73da547fd04557abd5aba73e76dd18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d73da547fd04557abd5aba73e76dd18 already in progress: creating tablet
W20251023 20:15:15.584791 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 799132aedc784de3bf0fc6478c8ef496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 799132aedc784de3bf0fc6478c8ef496 already in progress: creating tablet
W20251023 20:15:15.584991 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d8ad21e3888a4915ae5f252c2fca64cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ad21e3888a4915ae5f252c2fca64cb already in progress: creating tablet
I20251023 20:15:15.590989 23785 ts_tablet_manager.cc:1929] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.591157 23785 log.cc:1199] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/222ee7b9fa0a4bf69bb2ce8dab366f19
I20251023 20:15:15.591503 23785 ts_tablet_manager.cc:1950] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:15.593214 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cc5ce157b6ca42da90b051a184983bfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc5ce157b6ca42da90b051a184983bfe already in progress: creating tablet
I20251023 20:15:15.594727 23784 ts_tablet_manager.cc:1916] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:15.605413 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 73b4abd308054bac9d49b1d8e7dd2ebc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73b4abd308054bac9d49b1d8e7dd2ebc already in progress: creating tablet
W20251023 20:15:15.605865 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 799132aedc784de3bf0fc6478c8ef496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 799132aedc784de3bf0fc6478c8ef496 already in progress: creating tablet
W20251023 20:15:15.606062 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d8ad21e3888a4915ae5f252c2fca64cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ad21e3888a4915ae5f252c2fca64cb already in progress: creating tablet
W20251023 20:15:15.606283 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cc5ce157b6ca42da90b051a184983bfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc5ce157b6ca42da90b051a184983bfe already in progress: creating tablet
W20251023 20:15:15.606462 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 73b4abd308054bac9d49b1d8e7dd2ebc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73b4abd308054bac9d49b1d8e7dd2ebc already in progress: creating tablet
I20251023 20:15:15.606994 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e81dbc08f93d42d18c38eccb996633d4" } new_table_name: "default.Table_15d2e91d9bbd4fa8ac2260d9359076f4" modify_external_catalogs: true
I20251023 20:15:15.607331 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0d5856c857fb40e68632458d791c47ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.607517 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4f134fc02d1c4a53a726595803ef505c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.607838 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 (table default.Table_f01b7fbaa8924dfa965e4f6d3529dd2e [id=6e38fec2eebd48b8a6364be16ec99997]) successfully deleted
W20251023 20:15:15.607889 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d5856c857fb40e68632458d791c47ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:15.608022 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f134fc02d1c4a53a726595803ef505c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.608258 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.608582 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2bcd3241a40b49e2894f0104cebaad30" } new_table_name: "default.Table_a4375532322e46288e9b2a62d0a1cdea" modify_external_catalogs: true
I20251023 20:15:15.608757 23145 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6a1c8c9085ab4216b3dbfaa3a23ef61d" } new_table_name: "default.Table_537d89cd0e14498085731d83eae39d0a" modify_external_catalogs: true
I20251023 20:15:15.609273 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_91596859556249fbb6e82b7dc5756a92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.610176 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.610772 23786 ts_tablet_manager.cc:1434] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20251023 20:15:15.612571 23786 tablet_bootstrap.cc:492] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.612949 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet b09c0412f1eb48df8de542ca687a7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.613296 23783 tablet_replica.cc:333] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.613510 23783 raft_consensus.cc:2243] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.613642 23787 tablet_bootstrap.cc:492] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.613685 23783 raft_consensus.cc:2272] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.613801 23787 ts_tablet_manager.cc:1403] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.001s
W20251023 20:15:15.614046 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91596859556249fbb6e82b7dc5756a92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.614692 23783 ts_tablet_manager.cc:1916] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.614616 23787 raft_consensus.cc:359] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:15.614816 23787 raft_consensus.cc:385] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.614909 23787 raft_consensus.cc:740] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.615116 23787 consensus_queue.cc:260] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:15.616060 23787 ts_tablet_manager.cc:1434] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:15.616333 23787 tablet_bootstrap.cc:492] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.617710 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b2975466aafc47948a16bf139c0d70b1" } new_table_name: "default.Table_cae543cd317a44e4816ccc5a330b51a0" modify_external_catalogs: true
I20251023 20:15:15.620227 23787 tablet_bootstrap.cc:654] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.620326 23786 tablet_bootstrap.cc:654] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.622717 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet d9ebd537f271426a95af432ec9070ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:15.627302 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet cc5ce157b6ca42da90b051a184983bfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc5ce157b6ca42da90b051a184983bfe already in progress: creating tablet
W20251023 20:15:15.627951 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d8ad21e3888a4915ae5f252c2fca64cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ad21e3888a4915ae5f252c2fca64cb already in progress: creating tablet
I20251023 20:15:15.628069 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 (DEFAULT_TABLE table=default.Table_561d6afe24b246c9909547a248a5fe7f [id=a0f9058297034e739a638f1747ddacfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.628530 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487d9ee765d14aa2aa5668dc48fffe20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.630853 23787 tablet_bootstrap.cc:492] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.631011 23787 ts_tablet_manager.cc:1403] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20251023 20:15:15.631107 23786 tablet_bootstrap.cc:492] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.631244 23786 ts_tablet_manager.cc:1403] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20251023 20:15:15.631744 23787 raft_consensus.cc:359] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.631953 23787 raft_consensus.cc:385] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.632015 23787 raft_consensus.cc:740] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.631927 23786 raft_consensus.cc:359] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.632167 23786 raft_consensus.cc:385] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.632256 23786 raft_consensus.cc:740] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.632256 23787 consensus_queue.cc:260] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.632448 23786 consensus_queue.cc:260] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.635565 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 (DEFAULT_TABLE table=default.Table_561d6afe24b246c9909547a248a5fe7f [id=a0f9058297034e739a638f1747ddacfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.635565 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 (DEFAULT_TABLE table=default.Table_3320634019ac4bf48649eff33160d808 [id=c06c5925ec4e418e91167cd090df7f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.635934 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487d9ee765d14aa2aa5668dc48fffe20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.636147 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.636643 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 (DEFAULT_TABLE table=default.Table_561d6afe24b246c9909547a248a5fe7f [id=a0f9058297034e739a638f1747ddacfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.636996 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487d9ee765d14aa2aa5668dc48fffe20. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.642943 23786 ts_tablet_manager.cc:1434] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251023 20:15:15.643680 23790 tablet_bootstrap.cc:492] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.643894 23790 ts_tablet_manager.cc:1403] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.093s user 0.028s sys 0.010s
I20251023 20:15:15.644619 23786 tablet_bootstrap.cc:492] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.644778 23790 raft_consensus.cc:359] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.645037 23790 raft_consensus.cc:385] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.645160 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 (DEFAULT_TABLE table=default.Table_3320634019ac4bf48649eff33160d808 [id=c06c5925ec4e418e91167cd090df7f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.645555 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.646066 23790 raft_consensus.cc:740] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.646365 23786 tablet_bootstrap.cc:654] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.646394 23790 consensus_queue.cc:260] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:15.648356 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e44dfb4fd3d04a58b5fcc6102e055ff1 consensus state has no leader
W20251023 20:15:15.648660 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 0d73da547fd04557abd5aba73e76dd18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d73da547fd04557abd5aba73e76dd18 already in progress: creating tablet
W20251023 20:15:15.649633 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 73b4abd308054bac9d49b1d8e7dd2ebc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73b4abd308054bac9d49b1d8e7dd2ebc already in progress: creating tablet
I20251023 20:15:15.649652 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 60348d0c00c14247860b9df14ddb82b9 (DEFAULT_TABLE table=default.Table_baf231bb8eaa40eeb01169ddcf5161bd [id=94828b76eb8a4afdac4136fc626b6147]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.650012 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60348d0c00c14247860b9df14ddb82b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.651064 23379 tablet_service.cc:1505] Processing CreateTablet for tablet 60348d0c00c14247860b9df14ddb82b9 (DEFAULT_TABLE table=default.Table_baf231bb8eaa40eeb01169ddcf5161bd [id=94828b76eb8a4afdac4136fc626b6147]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.651407 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60348d0c00c14247860b9df14ddb82b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.653365 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 62acc395fb294965a1ab476d37f9fab1 (DEFAULT_TABLE table=default.Table_bf1bef05ff6749359d1c5be2beb1da6d [id=d199b1ef8bfe4b94963604bfffe6f2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.653689 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62acc395fb294965a1ab476d37f9fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.653666 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 60348d0c00c14247860b9df14ddb82b9 (DEFAULT_TABLE table=default.Table_baf231bb8eaa40eeb01169ddcf5161bd [id=94828b76eb8a4afdac4136fc626b6147]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.654036 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60348d0c00c14247860b9df14ddb82b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.655893 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 (DEFAULT_TABLE table=default.Table_3320634019ac4bf48649eff33160d808 [id=c06c5925ec4e418e91167cd090df7f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.658535 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 62acc395fb294965a1ab476d37f9fab1 (DEFAULT_TABLE table=default.Table_bf1bef05ff6749359d1c5be2beb1da6d [id=d199b1ef8bfe4b94963604bfffe6f2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.658905 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62acc395fb294965a1ab476d37f9fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.659161 23785 tablet_replica.cc:333] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.659327 23785 raft_consensus.cc:2243] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.659507 23785 raft_consensus.cc:2272] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:15.664014 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71263f7cf0994e78b5e7baa48fd4486b consensus state has no leader
I20251023 20:15:15.665616 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4d358925f1254129a8b42bfca11f8870"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.666191 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d358925f1254129a8b42bfca11f8870 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.667727 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_cbcc7255010047c288af86f302dfd125"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.668056 23783 ts_tablet_manager.cc:1929] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.668187 23783 log.cc:1199] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/b09c0412f1eb48df8de542ca687a7b48
W20251023 20:15:15.668216 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbcc7255010047c288af86f302dfd125 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.668536 23783 ts_tablet_manager.cc:1950] T b09c0412f1eb48df8de542ca687a7b48 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.668588 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_85297cf02a6d4a06a5931f0203962992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.669129 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85297cf02a6d4a06a5931f0203962992 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.669505 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet b09c0412f1eb48df8de542ca687a7b48 (table default.Table_45640565c4a24ff88274bc4e62b865ff [id=18ee017a8d484c7b9ff7dbbceaa5f7ac]) successfully deleted
I20251023 20:15:15.669872 23785 ts_tablet_manager.cc:1916] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.670034 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 62acc395fb294965a1ab476d37f9fab1 (DEFAULT_TABLE table=default.Table_bf1bef05ff6749359d1c5be2beb1da6d [id=d199b1ef8bfe4b94963604bfffe6f2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.670456 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62acc395fb294965a1ab476d37f9fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.670537 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b199143b364a43798ec81927ae2148ca" } new_table_name: "default.Table_2e1bf7e58ab84283b5f61b77865778ca" modify_external_catalogs: true
I20251023 20:15:15.670946 23784 ts_tablet_manager.cc:1929] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.671077 23784 log.cc:1199] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/bd41c9c84d814f9191d5952e71e31a06
I20251023 20:15:15.671417 23784 ts_tablet_manager.cc:1950] T bd41c9c84d814f9191d5952e71e31a06 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.671895 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet f49a0e08de024e3b97cb2e6ac5a59c9d from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.673235 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.673528 23783 tablet_replica.cc:333] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.673761 23783 raft_consensus.cc:2243] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.673843 23379 tablet_service.cc:1552] Processing DeleteTablet for tablet eedc0251bae548bfb4f1dd54f33b7a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.674044 23783 raft_consensus.cc:2272] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.674072 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet 24528de9d0a84eddb091dec4f1d2b4e2 from {username='slave'} at 127.0.0.1:40384
W20251023 20:15:15.674517 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 799132aedc784de3bf0fc6478c8ef496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 799132aedc784de3bf0fc6478c8ef496 already in progress: creating tablet
I20251023 20:15:15.674824 23783 ts_tablet_manager.cc:1916] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:15.675206 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 consensus state has no leader
I20251023 20:15:15.675491 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 56fd87a212ff4ce893a44597ec038be9 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.675562 23787 ts_tablet_manager.cc:1434] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251023 20:15:15.675827 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5
I20251023 20:15:15.676213 23787 tablet_bootstrap.cc:492] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.676268 23146 catalog_manager.cc:6719] ReplaceTablet: tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 deleted and replaced by tablet e31aebfc87734ffeb2c0b25c07cb3f70
I20251023 20:15:15.676512 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e31aebfc87734ffeb2c0b25c07cb3f70) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.676544 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_677942097b3a441287c448bb7c584a4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.677027 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_677942097b3a441287c448bb7c584a4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:15.678140 23145 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24528de9d0a84eddb091dec4f1d2b4e2 consensus state has no leader
I20251023 20:15:15.678831 23787 tablet_bootstrap.cc:654] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.679157 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e31aebfc87734ffeb2c0b25c07cb3f70) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.681037 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e31aebfc87734ffeb2c0b25c07cb3f70) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.685039 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f205f4e6d547229f3026c16a6fd485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.685372 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet bd41c9c84d814f9191d5952e71e31a06 (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]) successfully deleted
I20251023 20:15:15.687050 23784 tablet_replica.cc:333] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.687278 23784 raft_consensus.cc:2243] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.687686 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c3d2d37dd2bd414fae4c968fc3f52a25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.688081 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c from {username='slave'} at 127.0.0.1:40384
W20251023 20:15:15.688246 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3d2d37dd2bd414fae4c968fc3f52a25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.689070 23784 raft_consensus.cc:2272] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.690356 23784 ts_tablet_manager.cc:1916] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.693199 23801 raft_consensus.cc:493] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:15.693495 23801 raft_consensus.cc:515] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:15.696952 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1685213b588148febd6a23e1d2d6691d consensus state has no leader
I20251023 20:15:15.697062 23801 leader_election.cc:290] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:15.697743 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 24528de9d0a84eddb091dec4f1d2b4e2
I20251023 20:15:15.697999 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6f178deaeeb94c4da6a40568c635e0b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.698194 23148 catalog_manager.cc:6719] ReplaceTablet: tablet 24528de9d0a84eddb091dec4f1d2b4e2 deleted and replaced by tablet 451955aee3aa4bcb82252bcefbbcaefe
W20251023 20:15:15.698534 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f178deaeeb94c4da6a40568c635e0b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.699018 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 451955aee3aa4bcb82252bcefbbcaefe) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.699323 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 451955aee3aa4bcb82252bcefbbcaefe) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.699828 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 451955aee3aa4bcb82252bcefbbcaefe) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.703805 23783 ts_tablet_manager.cc:1929] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.703936 23783 log.cc:1199] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d9ebd537f271426a95af432ec9070ae5
I20251023 20:15:15.704286 23783 ts_tablet_manager.cc:1950] T d9ebd537f271426a95af432ec9070ae5 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.704954 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet f49a0e08de024e3b97cb2e6ac5a59c9d
I20251023 20:15:15.705291 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.705427 23139 catalog_manager.cc:6719] ReplaceTablet: tablet f49a0e08de024e3b97cb2e6ac5a59c9d deleted and replaced by tablet 6558265ff9e249c68123577958f1401a
I20251023 20:15:15.705806 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d with delete_type TABLET_DATA_DELETED (replaced by tablet 6558265ff9e249c68123577958f1401a) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.706380 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d with delete_type TABLET_DATA_DELETED (replaced by tablet 6558265ff9e249c68123577958f1401a) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.708508 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d9ebd537f271426a95af432ec9070ae5 (table default.Table_bbb88a1064eb4a27aa71756ff68bb6db [id=197fa361f58a4b24b5b394eb7f43b4e9]) successfully deleted
I20251023 20:15:15.708501 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d05145bbb50343f38166de6b35bad890" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:15.708866 23783 tablet_replica.cc:333] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:15.709132 23605 leader_election.cc:343] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:15.709483 23783 raft_consensus.cc:2243] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.709847 23783 raft_consensus.cc:2272] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.711454 23790 ts_tablet_manager.cc:1434] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20251023 20:15:15.712903 23790 tablet_bootstrap.cc:492] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.713233 23783 ts_tablet_manager.cc:1916] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.714584 23790 tablet_bootstrap.cc:654] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.715718 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d with delete_type TABLET_DATA_DELETED (replaced by tablet 6558265ff9e249c68123577958f1401a) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.718542 23785 ts_tablet_manager.cc:1929] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.721920 23785 log.cc:1199] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d9b25166b00842948980cdfcacdd5df2
I20251023 20:15:15.722983 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c
I20251023 20:15:15.723496 23140 catalog_manager.cc:6719] ReplaceTablet: tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c deleted and replaced by tablet 8e19f64a9ac44efba92928d0fa16819a
I20251023 20:15:15.724244 23785 ts_tablet_manager.cc:1950] T d9b25166b00842948980cdfcacdd5df2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:15.724541 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c with delete_type TABLET_DATA_DELETED (replaced by tablet 8e19f64a9ac44efba92928d0fa16819a) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.724864 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c with delete_type TABLET_DATA_DELETED (replaced by tablet 8e19f64a9ac44efba92928d0fa16819a) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.723690 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c with delete_type TABLET_DATA_DELETED (replaced by tablet 8e19f64a9ac44efba92928d0fa16819a) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.719394 23790 tablet_bootstrap.cc:492] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.725394 23790 ts_tablet_manager.cc:1403] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251023 20:15:15.725517 23786 tablet_bootstrap.cc:492] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.725697 23786 ts_tablet_manager.cc:1403] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.004s
I20251023 20:15:15.726408 23790 raft_consensus.cc:359] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.726441 23786 raft_consensus.cc:359] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.726629 23786 raft_consensus.cc:385] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.726701 23786 raft_consensus.cc:740] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.726962 23786 consensus_queue.cc:260] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.727372 23790 raft_consensus.cc:385] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.728128 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d9b25166b00842948980cdfcacdd5df2 (table default.Table_f66ae376b73c46d180bde83fb80dc7bb [id=4f40834f1fae488bb070b0c40c191ee8]) successfully deleted
I20251023 20:15:15.729159 23790 raft_consensus.cc:740] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.729393 23786 ts_tablet_manager.cc:1434] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:15.729733 23790 consensus_queue.cc:260] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.731839 23786 tablet_bootstrap.cc:492] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.730268 23785 tablet_replica.cc:333] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.735483 23785 raft_consensus.cc:2243] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.736025 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.736075 23783 ts_tablet_manager.cc:1929] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.736210 23783 log.cc:1199] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/6ae1876fa334495ca95dc2e9e6b4a0fd
I20251023 20:15:15.736534 23783 ts_tablet_manager.cc:1950] T 6ae1876fa334495ca95dc2e9e6b4a0fd P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.735877 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 24528de9d0a84eddb091dec4f1d2b4e2) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.737684 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_cae543cd317a44e4816ccc5a330b51a0" } modify_external_catalogs: true
I20251023 20:15:15.737882 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_cae543cd317a44e4816ccc5a330b51a0" } modify_external_catalogs: true
I20251023 20:15:15.738183 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_baf231bb8eaa40eeb01169ddcf5161bd" } modify_external_catalogs: true
I20251023 20:15:15.738363 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_baf231bb8eaa40eeb01169ddcf5161bd" } modify_external_catalogs: true
I20251023 20:15:15.738458 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fb1a22feead44f33b0a8f5f5ec849493" } modify_external_catalogs: true
I20251023 20:15:15.738641 23786 tablet_bootstrap.cc:654] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.738941 23784 ts_tablet_manager.cc:1929] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.738937 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_853f1904ea4d44919b14dc452f631d2f" } modify_external_catalogs: true
I20251023 20:15:15.739200 23785 raft_consensus.cc:2272] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.739259 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_853f1904ea4d44919b14dc452f631d2f" } modify_external_catalogs: true
I20251023 20:15:15.739320 23790 ts_tablet_manager.cc:1434] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:15.738637 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fb1a22feead44f33b0a8f5f5ec849493" } modify_external_catalogs: true
I20251023 20:15:15.741852 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2e1bf7e58ab84283b5f61b77865778ca" } modify_external_catalogs: true
I20251023 20:15:15.742075 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2e1bf7e58ab84283b5f61b77865778ca" } modify_external_catalogs: true
I20251023 20:15:15.737453 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 6ae1876fa334495ca95dc2e9e6b4a0fd (table default.Table_8b24573ba4864a13a123e5d8d2bf9448 [id=fb7af3cee68a4f3c9307a5876db279c1]) successfully deleted
W20251023 20:15:15.744927 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 already in progress: creating tablet
I20251023 20:15:15.745203 23787 tablet_bootstrap.cc:492] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.745520 23787 ts_tablet_manager.cc:1403] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.003s
W20251023 20:15:15.745757 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 24528de9d0a84eddb091dec4f1d2b4e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24528de9d0a84eddb091dec4f1d2b4e2 already in progress: creating tablet
W20251023 20:15:15.746345 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f49a0e08de024e3b97cb2e6ac5a59c9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f49a0e08de024e3b97cb2e6ac5a59c9d already in progress: creating tablet
I20251023 20:15:15.746438 23787 raft_consensus.cc:359] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.746632 23787 raft_consensus.cc:385] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.746727 23787 raft_consensus.cc:740] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:15.747014 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c already in progress: creating tablet
I20251023 20:15:15.746959 23787 consensus_queue.cc:260] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.748184 23787 ts_tablet_manager.cc:1434] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:15.748765 23787 tablet_bootstrap.cc:492] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.749410 23790 tablet_bootstrap.cc:492] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.752489 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet fcec61131b7f4af590acf88a2ab3c5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.752799 23783 tablet_replica.cc:333] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.752986 23783 raft_consensus.cc:2243] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.753232 23783 raft_consensus.cc:2272] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.753908 23783 ts_tablet_manager.cc:1916] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.750142 23785 ts_tablet_manager.cc:1916] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.757804 23787 tablet_bootstrap.cc:654] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.758595 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 1685213b588148febd6a23e1d2d6691d
I20251023 20:15:15.743501 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d05145bbb50343f38166de6b35bad890" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:15.759665 23784 log.cc:1199] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/eedc0251bae548bfb4f1dd54f33b7a86
I20251023 20:15:15.759956 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685213b588148febd6a23e1d2d6691d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.760330 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685213b588148febd6a23e1d2d6691d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.760500 23784 ts_tablet_manager.cc:1950] T eedc0251bae548bfb4f1dd54f33b7a86 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.760967 23790 tablet_bootstrap.cc:654] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:15.759500 23604 leader_election.cc:343] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:15.759941 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685213b588148febd6a23e1d2d6691d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.761157 23604 leader_election.cc:304] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:15.761927 23801 raft_consensus.cc:2749] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:15.764510 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet eedc0251bae548bfb4f1dd54f33b7a86 (table default.Table_c747a4f8ca7a461e90896f4145d9ceb6 [id=08f1b14fc24d49baa3869370048151aa]) successfully deleted
I20251023 20:15:15.763373 23378 tablet_service.cc:1505] Processing CreateTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 (DEFAULT_TABLE table=default.Table_fb1a22feead44f33b0a8f5f5ec849493 [id=d05778ff4e54433d9d549b7d5356afe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.765470 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caaccec5866a4e95afe9ccbf1f5bc494. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.765547 23786 tablet_bootstrap.cc:492] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.765697 23786 ts_tablet_manager.cc:1403] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.002s
I20251023 20:15:15.766813 23786 raft_consensus.cc:359] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.767012 23786 raft_consensus.cc:385] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.767099 23786 raft_consensus.cc:740] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.767315 23786 consensus_queue.cc:260] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.767761 23505 tablet_service.cc:1505] Processing CreateTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 (DEFAULT_TABLE table=default.Table_fb1a22feead44f33b0a8f5f5ec849493 [id=d05778ff4e54433d9d549b7d5356afe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.768172 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caaccec5866a4e95afe9ccbf1f5bc494. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.770129 23786 ts_tablet_manager.cc:1434] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:15.772146 23786 tablet_bootstrap.cc:492] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.773661 23783 ts_tablet_manager.cc:1929] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.773865 23783 log.cc:1199] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/fcec61131b7f4af590acf88a2ab3c5f1
I20251023 20:15:15.774148 23786 tablet_bootstrap.cc:654] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.774340 23783 ts_tablet_manager.cc:1950] T fcec61131b7f4af590acf88a2ab3c5f1 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.775298 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet fcec61131b7f4af590acf88a2ab3c5f1 (table default.Table_07bcc06b3cdd42379508c3f649756c40 [id=0d5215e658b64dbcbc9d37fc2622ca61]) successfully deleted
I20251023 20:15:15.775693 23637 tablet_service.cc:1505] Processing CreateTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 (DEFAULT_TABLE table=default.Table_fb1a22feead44f33b0a8f5f5ec849493 [id=d05778ff4e54433d9d549b7d5356afe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.776057 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caaccec5866a4e95afe9ccbf1f5bc494. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.776468 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 already in progress: creating tablet
I20251023 20:15:15.778023 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 60348d0c00c14247860b9df14ddb82b9
I20251023 20:15:15.778070 23784 tablet_replica.cc:333] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.778239 23784 raft_consensus.cc:2243] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.778391 23784 raft_consensus.cc:2272] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.778926 23378 tablet_service.cc:1505] Processing CreateTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f (DEFAULT_TABLE table=default.Table_0d5856c857fb40e68632458d791c47ba [id=b0b39a6a41394aaab2423e84baa0a063]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.779096 23784 ts_tablet_manager.cc:1916] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.779106 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 60348d0c00c14247860b9df14ddb82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.779228 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 71263f7cf0994e78b5e7baa48fd4486b
I20251023 20:15:15.779299 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bb6abe3ef848e2831839de6a28eb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.780059 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.780304 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.781378 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 60348d0c00c14247860b9df14ddb82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.781617 23379 tablet_service.cc:1552] Processing DeleteTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:15.783767 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1685213b588148febd6a23e1d2d6691d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1685213b588148febd6a23e1d2d6691d already in progress: creating tablet
W20251023 20:15:15.784394 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 60348d0c00c14247860b9df14ddb82b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60348d0c00c14247860b9df14ddb82b9 already in progress: creating tablet
W20251023 20:15:15.784919 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 71263f7cf0994e78b5e7baa48fd4486b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71263f7cf0994e78b5e7baa48fd4486b already in progress: creating tablet
I20251023 20:15:15.786024 23501 tablet_service.cc:1505] Processing CreateTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f (DEFAULT_TABLE table=default.Table_0d5856c857fb40e68632458d791c47ba [id=b0b39a6a41394aaab2423e84baa0a063]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.786401 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bb6abe3ef848e2831839de6a28eb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.786686 23790 tablet_bootstrap.cc:492] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.786844 23790 ts_tablet_manager.cc:1403] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20251023 20:15:15.787614 23790 raft_consensus.cc:359] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.787806 23790 raft_consensus.cc:385] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.787885 23790 raft_consensus.cc:740] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.790848 23785 ts_tablet_manager.cc:1929] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.794682 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 60348d0c00c14247860b9df14ddb82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.795013 23785 log.cc:1199] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/eedc0251bae548bfb4f1dd54f33b7a86
I20251023 20:15:15.795778 23787 tablet_bootstrap.cc:492] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.796881 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d8aa8677f0ad4af6ae46b9ce731a7095
I20251023 20:15:15.798295 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet caaccec5866a4e95afe9ccbf1f5bc494
I20251023 20:15:15.799621 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.800034 23786 tablet_bootstrap.cc:492] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.800208 23786 ts_tablet_manager.cc:1403] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20251023 20:15:15.801396 23786 raft_consensus.cc:359] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.801607 23786 raft_consensus.cc:385] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.801678 23786 raft_consensus.cc:740] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.801883 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.801961 23786 consensus_queue.cc:260] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.804446 23786 ts_tablet_manager.cc:1434] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:15.797995 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.804487 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 (DEFAULT_TABLE table=default.Table_4f134fc02d1c4a53a726595803ef505c [id=ff3dd4d1748144c492b64ccde7297729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.799197 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.804877 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e338b982a564ea5b1b15c2146a96fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.804950 23787 ts_tablet_manager.cc:1403] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.001s
I20251023 20:15:15.805562 23786 tablet_bootstrap.cc:492] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.803097 23650 tablet_service.cc:1505] Processing CreateTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f (DEFAULT_TABLE table=default.Table_0d5856c857fb40e68632458d791c47ba [id=b0b39a6a41394aaab2423e84baa0a063]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.805807 23787 raft_consensus.cc:359] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.805996 23787 raft_consensus.cc:385] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.806077 23787 raft_consensus.cc:740] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.806105 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4bb6abe3ef848e2831839de6a28eb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.806331 23787 consensus_queue.cc:260] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.807525 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.807793 23786 tablet_bootstrap.cc:654] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.808794 23784 ts_tablet_manager.cc:1929] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.808905 23784 log.cc:1199] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/73f205f4e6d547229f3026c16a6fd485
I20251023 20:15:15.809252 23784 ts_tablet_manager.cc:1950] T 73f205f4e6d547229f3026c16a6fd485 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.809556 23787 ts_tablet_manager.cc:1434] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251023 20:15:15.809873 23787 tablet_bootstrap.cc:492] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.810788 23784 tablet_replica.cc:333] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.810981 23784 raft_consensus.cc:2243] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.811173 23784 raft_consensus.cc:2272] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.811378 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 73f205f4e6d547229f3026c16a6fd485 (table default.Table_43bf18200b234de68377af27f882f395 [id=da87b3d063a64aab92f939df02727144]) successfully deleted
I20251023 20:15:15.811444 23787 tablet_bootstrap.cc:654] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:15.811650 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d8aa8677f0ad4af6ae46b9ce731a7095 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8aa8677f0ad4af6ae46b9ce731a7095 already in progress: creating tablet
I20251023 20:15:15.811635 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 (DEFAULT_TABLE table=default.Table_4f134fc02d1c4a53a726595803ef505c [id=ff3dd4d1748144c492b64ccde7297729]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:15.811867 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet caaccec5866a4e95afe9ccbf1f5bc494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caaccec5866a4e95afe9ccbf1f5bc494 already in progress: creating tablet
I20251023 20:15:15.812115 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e338b982a564ea5b1b15c2146a96fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.812192 23784 ts_tablet_manager.cc:1916] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.800881 23379 tablet_service.cc:1552] Processing DeleteTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.801047 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.803929 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 (DEFAULT_TABLE table=default.Table_4f134fc02d1c4a53a726595803ef505c [id=ff3dd4d1748144c492b64ccde7297729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.813350 23785 ts_tablet_manager.cc:1950] T eedc0251bae548bfb4f1dd54f33b7a86 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:15.813537 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e338b982a564ea5b1b15c2146a96fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.813480 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_05b7b64aa5754971b4754610122a411b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.813993 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b7b64aa5754971b4754610122a411b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.791113 23790 consensus_queue.cc:260] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.814597 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.815043 23783 tablet_replica.cc:333] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.815205 23783 raft_consensus.cc:2243] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.815351 23783 raft_consensus.cc:2272] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.816083 23783 ts_tablet_manager.cc:1916] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.817775 23790 ts_tablet_manager.cc:1434] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.031s user 0.002s sys 0.001s
I20251023 20:15:15.818987 23790 tablet_bootstrap.cc:492] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.821192 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 2177416169ca4d0d8368b1cac90c513e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.825515 23790 tablet_bootstrap.cc:654] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.828033 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet eedc0251bae548bfb4f1dd54f33b7a86 (table default.Table_c747a4f8ca7a461e90896f4145d9ceb6 [id=08f1b14fc24d49baa3869370048151aa]) successfully deleted
W20251023 20:15:15.828688 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 already in progress: creating tablet
W20251023 20:15:15.829300 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 24528de9d0a84eddb091dec4f1d2b4e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24528de9d0a84eddb091dec4f1d2b4e2 already in progress: creating tablet
W20251023 20:15:15.830775 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet f49a0e08de024e3b97cb2e6ac5a59c9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f49a0e08de024e3b97cb2e6ac5a59c9d already in progress: creating tablet
W20251023 20:15:15.831369 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c already in progress: creating tablet
I20251023 20:15:15.831722 23786 tablet_bootstrap.cc:492] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.831887 23786 ts_tablet_manager.cc:1403] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20251023 20:15:15.832667 23786 raft_consensus.cc:359] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.832861 23786 raft_consensus.cc:385] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.832955 23786 raft_consensus.cc:740] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.833271 23785 tablet_replica.cc:333] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.833230 23786 consensus_queue.cc:260] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.833575 23785 raft_consensus.cc:2243] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.834518 23785 raft_consensus.cc:2272] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.835772 23786 ts_tablet_manager.cc:1434] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:15.836651 23785 ts_tablet_manager.cc:1916] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.837323 23786 tablet_bootstrap.cc:492] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.838811 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet c3035573f53c4a6db0bb72882ca84f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.838938 23783 ts_tablet_manager.cc:1929] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.839218 23783 log.cc:1199] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/56fd87a212ff4ce893a44597ec038be9
I20251023 20:15:15.839843 23786 tablet_bootstrap.cc:654] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.841583 23783 ts_tablet_manager.cc:1950] T 56fd87a212ff4ce893a44597ec038be9 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.842571 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 56fd87a212ff4ce893a44597ec038be9 (table default.Table_0acacb2edf1c461baf55cc00e9a69ff6 [id=8478b83c0ee645fba6c41151e56246b7]) successfully deleted
I20251023 20:15:15.843362 23784 ts_tablet_manager.cc:1929] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.844282 23784 log.cc:1199] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/24528de9d0a84eddb091dec4f1d2b4e2
I20251023 20:15:15.844754 23784 ts_tablet_manager.cc:1950] T 24528de9d0a84eddb091dec4f1d2b4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.845749 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 24528de9d0a84eddb091dec4f1d2b4e2 (table default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
W20251023 20:15:15.846392 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f49a0e08de024e3b97cb2e6ac5a59c9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f49a0e08de024e3b97cb2e6ac5a59c9d already in progress: creating tablet
W20251023 20:15:15.847047 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c already in progress: creating tablet
I20251023 20:15:15.847452 23784 tablet_replica.cc:333] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.847874 23784 raft_consensus.cc:2243] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.848552 23784 raft_consensus.cc:2272] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.849644 23784 ts_tablet_manager.cc:1916] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.850335 23786 tablet_bootstrap.cc:492] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.850606 23786 ts_tablet_manager.cc:1403] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20251023 20:15:15.851405 23786 raft_consensus.cc:359] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.851673 23786 raft_consensus.cc:385] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.851780 23786 raft_consensus.cc:740] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.852039 23787 tablet_bootstrap.cc:492] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.852077 23786 consensus_queue.cc:260] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.852178 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_1662565661d643d8b4ff35b3e84f0351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.852749 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1662565661d643d8b4ff35b3e84f0351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.853135 23790 tablet_bootstrap.cc:492] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.853354 23790 ts_tablet_manager.cc:1403] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20251023 20:15:15.854183 23790 raft_consensus.cc:359] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.854452 23787 ts_tablet_manager.cc:1403] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.003s
I20251023 20:15:15.855264 23787 raft_consensus.cc:359] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.855540 23790 raft_consensus.cc:385] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.855662 23790 raft_consensus.cc:740] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.856011 23790 consensus_queue.cc:260] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.856264 23787 raft_consensus.cc:385] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.856475 23787 raft_consensus.cc:740] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.856765 23786 ts_tablet_manager.cc:1434] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:15.857867 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.858852 23785 ts_tablet_manager.cc:1929] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.859448 23786 tablet_bootstrap.cc:492] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.861053 23786 tablet_bootstrap.cc:654] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.862191 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 364e3b9a07e44c8992f6d9cbda6a86e5 (DEFAULT_TABLE table=default.Table_677942097b3a441287c448bb7c584a4f [id=7fa58e925ef94309ba20deab24c9be99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.862852 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364e3b9a07e44c8992f6d9cbda6a86e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.863271 23787 consensus_queue.cc:260] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.863559 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet b125422cf17145d5aef418817082eb32 with delete_type TABLET_DATA_DELETED (replaced by tablet e44dfb4fd3d04a58b5fcc6102e055ff1) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.863822 23785 log.cc:1199] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/9a2d1315afa0468c956adf64e1bd9cf0
I20251023 20:15:15.864266 23785 ts_tablet_manager.cc:1950] T 9a2d1315afa0468c956adf64e1bd9cf0 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:15.866127 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_24d0ef155c144a43828c4de4e53e0c00" } new_table_name: "default.Table_363dbff3b2a046b190cdf7ce1a8572d6" modify_external_catalogs: true
I20251023 20:15:15.866217 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 364e3b9a07e44c8992f6d9cbda6a86e5 (DEFAULT_TABLE table=default.Table_677942097b3a441287c448bb7c584a4f [id=7fa58e925ef94309ba20deab24c9be99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.866575 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364e3b9a07e44c8992f6d9cbda6a86e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.867453 23784 ts_tablet_manager.cc:1929] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.867607 23784 log.cc:1199] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/fcec61131b7f4af590acf88a2ab3c5f1
I20251023 20:15:15.867973 23784 ts_tablet_manager.cc:1950] T fcec61131b7f4af590acf88a2ab3c5f1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.865119 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 364e3b9a07e44c8992f6d9cbda6a86e5 (DEFAULT_TABLE table=default.Table_677942097b3a441287c448bb7c584a4f [id=7fa58e925ef94309ba20deab24c9be99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.868908 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet fcec61131b7f4af590acf88a2ab3c5f1 (table default.Table_07bcc06b3cdd42379508c3f649756c40 [id=0d5215e658b64dbcbc9d37fc2622ca61]) successfully deleted
I20251023 20:15:15.869125 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364e3b9a07e44c8992f6d9cbda6a86e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.871284 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 040a44e272b7410c98c109fbceb19192 (DEFAULT_TABLE table=default.Table_cbcc7255010047c288af86f302dfd125 [id=70afc97d36fa4bf0a81c6b6611d3975a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.871677 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 040a44e272b7410c98c109fbceb19192. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.872231 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0232c33db59641b3af27bfd15cd7fb81 consensus state has no leader
W20251023 20:15:15.873791 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1685213b588148febd6a23e1d2d6691d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1685213b588148febd6a23e1d2d6691d already in progress: creating tablet
W20251023 20:15:15.874774 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 71263f7cf0994e78b5e7baa48fd4486b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71263f7cf0994e78b5e7baa48fd4486b already in progress: creating tablet
I20251023 20:15:15.868701 23790 ts_tablet_manager.cc:1434] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251023 20:15:15.875705 23786 tablet_bootstrap.cc:492] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.875908 23786 ts_tablet_manager.cc:1403] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20251023 20:15:15.878096 23790 tablet_bootstrap.cc:492] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.879602 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f01b7fbaa8924dfa965e4f6d3529dd2e" } new_table_name: "default.Table_b6760837eeee4b5082522107b87d72f5" modify_external_catalogs: true
I20251023 20:15:15.879884 23787 ts_tablet_manager.cc:1434] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251023 20:15:15.882081 23787 tablet_bootstrap.cc:492] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.876776 23786 raft_consensus.cc:359] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.883128 23786 raft_consensus.cc:385] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.883231 23786 raft_consensus.cc:740] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.883503 23786 consensus_queue.cc:260] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.883833 23787 tablet_bootstrap.cc:654] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.885571 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6cf378e7dc9b4a37acaec754f2edc359"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.885665 23794 raft_consensus.cc:493] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:15.885808 23794 raft_consensus.cc:515] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:15.886143 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cf378e7dc9b4a37acaec754f2edc359 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.886227 23790 tablet_bootstrap.cc:654] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.886355 23794 leader_election.cc:290] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:15.887233 23786 ts_tablet_manager.cc:1434] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251023 20:15:15.887574 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:15.887964 23470 leader_election.cc:343] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 536216374bae4bc481d78fabbed3567f
I20251023 20:15:15.888229 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "536216374bae4bc481d78fabbed3567f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:15.889238 23471 leader_election.cc:343] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:15.889377 23786 tablet_bootstrap.cc:492] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.889379 23471 leader_election.cc:304] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:15.889636 23794 raft_consensus.cc:2749] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:15.890720 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet 56fd87a212ff4ce893a44597ec038be9 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.891173 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_05b7b64aa5754971b4754610122a411b" } new_table_name: "default.Table_11159791d68d4dfb98fe2e8d7bb3afdc" modify_external_catalogs: true
I20251023 20:15:15.891356 23786 tablet_bootstrap.cc:654] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.891325 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_9dd11819a7d24778856f462839d28a31" } new_table_name: "default.Table_e58c00cb57f34786864684583ab1adf6" modify_external_catalogs: true
W20251023 20:15:15.892652 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 60348d0c00c14247860b9df14ddb82b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60348d0c00c14247860b9df14ddb82b9 already in progress: creating tablet
I20251023 20:15:15.892831 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_67a7910474894d659750844c77625a9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.893384 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67a7910474894d659750844c77625a9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:15.893625 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet caaccec5866a4e95afe9ccbf1f5bc494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caaccec5866a4e95afe9ccbf1f5bc494 already in progress: creating tablet
W20251023 20:15:15.894253 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d8aa8677f0ad4af6ae46b9ce731a7095 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8aa8677f0ad4af6ae46b9ce731a7095 already in progress: creating tablet
I20251023 20:15:15.894546 23784 tablet_replica.cc:333] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.894728 23784 raft_consensus.cc:2243] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.895092 23784 raft_consensus.cc:2272] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.896060 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 9a2d1315afa0468c956adf64e1bd9cf0 (table default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
W20251023 20:15:15.897228 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1685213b588148febd6a23e1d2d6691d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1685213b588148febd6a23e1d2d6691d already in progress: creating tablet
I20251023 20:15:15.897205 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 040a44e272b7410c98c109fbceb19192 (DEFAULT_TABLE table=default.Table_cbcc7255010047c288af86f302dfd125 [id=70afc97d36fa4bf0a81c6b6611d3975a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.897264 23784 ts_tablet_manager.cc:1916] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.897701 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 040a44e272b7410c98c109fbceb19192. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.899715 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_58731fce9c224ff0b59d4b241110e518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.899798 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bf1bef05ff6749359d1c5be2beb1da6d" } new_table_name: "default.Table_bd80c8158a014c8192a0800e8c1c32b6" modify_external_catalogs: true
W20251023 20:15:15.900281 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58731fce9c224ff0b59d4b241110e518 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.901333 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 040a44e272b7410c98c109fbceb19192 (DEFAULT_TABLE table=default.Table_cbcc7255010047c288af86f302dfd125 [id=70afc97d36fa4bf0a81c6b6611d3975a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.901706 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 040a44e272b7410c98c109fbceb19192. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.901885 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5416da30c66645c19d36112980608527 consensus state has no leader
I20251023 20:15:15.902941 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 with delete_type TABLET_DATA_DELETED (replaced by tablet 5416da30c66645c19d36112980608527) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.903198 23783 tablet_replica.cc:333] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:15.903349 23783 raft_consensus.cc:2243] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.903519 23783 raft_consensus.cc:2272] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.904331 23783 ts_tablet_manager.cc:1916] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:15.908919 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e19f64a9ac44efba92928d0fa16819a consensus state has no leader
W20251023 20:15:15.909495 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e76f2e26dcd4cdfbf72de899f5a1458 has no consensus state
I20251023 20:15:15.912331 23783 ts_tablet_manager.cc:1929] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.912731 23783 log.cc:1199] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/222ee7b9fa0a4bf69bb2ce8dab366f19
I20251023 20:15:15.913247 23783 ts_tablet_manager.cc:1950] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.913707 23786 tablet_bootstrap.cc:492] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.913954 23786 ts_tablet_manager.cc:1403] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20251023 20:15:15.914265 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 (table default.Table_b6760837eeee4b5082522107b87d72f5 [id=6e38fec2eebd48b8a6364be16ec99997]) successfully deleted
I20251023 20:15:15.914821 23786 raft_consensus.cc:359] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.915104 23786 raft_consensus.cc:385] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.915210 23786 raft_consensus.cc:740] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.915460 23786 consensus_queue.cc:260] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.917641 23786 ts_tablet_manager.cc:1434] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:15.918242 23786 tablet_bootstrap.cc:492] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.920617 23786 tablet_bootstrap.cc:654] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.906755 23784 ts_tablet_manager.cc:1929] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.921703 23784 log.cc:1199] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/216e2c887a364c8ea5068c3834d1e4e2
W20251023 20:15:15.922024 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62acc395fb294965a1ab476d37f9fab1 consensus state has no leader
I20251023 20:15:15.922396 23784 ts_tablet_manager.cc:1950] T 216e2c887a364c8ea5068c3834d1e4e2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.923415 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 216e2c887a364c8ea5068c3834d1e4e2 (table default.Table_d94baddda6104b5c9c007c95623ea0dd [id=ce02fa77823e4fbcad36a7e21bf10630]) successfully deleted
I20251023 20:15:15.924010 23784 tablet_replica.cc:333] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.924255 23784 raft_consensus.cc:2243] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.924467 23784 raft_consensus.cc:2272] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.925251 23784 ts_tablet_manager.cc:1916] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.931596 23787 tablet_bootstrap.cc:492] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:15.931613 23784 ts_tablet_manager.cc:1929] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.931742 23784 log.cc:1199] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/c3035573f53c4a6db0bb72882ca84f37
I20251023 20:15:15.933745 23787 ts_tablet_manager.cc:1403] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20251023 20:15:15.934063 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0898b76943db42e98d75a6e39cb8fade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.934309 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_82f6a78855d1488b9a8085c3bfbebab3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.934585 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_eb3f472232804ea98a49fe822c8e9118"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.934777 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0898b76943db42e98d75a6e39cb8fade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:15.934845 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82f6a78855d1488b9a8085c3bfbebab3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:15.934954 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb3f472232804ea98a49fe822c8e9118 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.935503 23786 tablet_bootstrap.cc:492] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.935698 23786 ts_tablet_manager.cc:1403] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20251023 20:15:15.935917 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet e44dfb4fd3d04a58b5fcc6102e055ff1 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.936532 23786 raft_consensus.cc:359] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.936712 23786 raft_consensus.cc:385] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.936929 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 0d73da547fd04557abd5aba73e76dd18 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.937186 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet cc5ce157b6ca42da90b051a184983bfe from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.937565 23786 raft_consensus.cc:740] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.937788 23786 consensus_queue.cc:260] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.938886 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_bb6092ff59da4b9988c720c4987d7c35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:15.939131 23786 ts_tablet_manager.cc:1434] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:15.939422 23786 tablet_bootstrap.cc:492] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:15.939396 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb6092ff59da4b9988c720c4987d7c35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.939968 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_265f006bfcc2450f964410c799f035f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:15.940433 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_265f006bfcc2450f964410c799f035f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:15.941356 23786 tablet_bootstrap.cc:654] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.942621 23784 ts_tablet_manager.cc:1950] T c3035573f53c4a6db0bb72882ca84f37 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.943142 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet c3035573f53c4a6db0bb72882ca84f37 (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:15.943568 23784 tablet_replica.cc:333] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.943810 23784 raft_consensus.cc:2243] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.944085 23784 raft_consensus.cc:2272] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.944926 23784 ts_tablet_manager.cc:1916] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.946310 23790 tablet_bootstrap.cc:492] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:15.946480 23790 ts_tablet_manager.cc:1403] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.000s
I20251023 20:15:15.947294 23790 raft_consensus.cc:359] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.948884 23790 raft_consensus.cc:385] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.949141 23790 raft_consensus.cc:740] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.949566 23790 consensus_queue.cc:260] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.950201 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 with delete_type TABLET_DATA_DELETED (replaced by tablet 5416da30c66645c19d36112980608527) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.950429 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 64f515399c494aadbed2bd0eac171a77 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:15.950578 23787 raft_consensus.cc:359] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.950775 23787 raft_consensus.cc:385] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.950846 23787 raft_consensus.cc:740] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:15.954757 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 60348d0c00c14247860b9df14ddb82b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60348d0c00c14247860b9df14ddb82b9 already in progress: creating tablet
W20251023 20:15:15.955055 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 71263f7cf0994e78b5e7baa48fd4486b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71263f7cf0994e78b5e7baa48fd4486b already in progress: creating tablet
W20251023 20:15:15.955438 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d8aa8677f0ad4af6ae46b9ce731a7095 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8aa8677f0ad4af6ae46b9ce731a7095 already in progress: creating tablet
I20251023 20:15:15.955633 23784 ts_tablet_manager.cc:1929] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:15.956001 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet caaccec5866a4e95afe9ccbf1f5bc494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caaccec5866a4e95afe9ccbf1f5bc494 already in progress: creating tablet
I20251023 20:15:15.956128 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 7f766bfe7dd94352abcaf34a200d00bc (DEFAULT_TABLE table=default.Table_85297cf02a6d4a06a5931f0203962992 [id=3289908651fc4a63ba05b7b30c2ed718]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.956598 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f766bfe7dd94352abcaf34a200d00bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.957967 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet b09c0412f1eb48df8de542ca687a7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.959107 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ae1876fa334495ca95dc2e9e6b4a0fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.959499 23784 log.cc:1199] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/56fd87a212ff4ce893a44597ec038be9
I20251023 20:15:15.960053 23784 ts_tablet_manager.cc:1950] T 56fd87a212ff4ce893a44597ec038be9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:15.960533 23785 tablet_replica.cc:333] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:15.960987 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_39878ca4314e43b8bf37b4cba1a5fb54" } modify_external_catalogs: true
I20251023 20:15:15.961189 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_39878ca4314e43b8bf37b4cba1a5fb54" } modify_external_catalogs: true
I20251023 20:15:15.961462 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 56fd87a212ff4ce893a44597ec038be9 (table default.Table_0acacb2edf1c461baf55cc00e9a69ff6 [id=8478b83c0ee645fba6c41151e56246b7]) successfully deleted
I20251023 20:15:15.963042 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 7f766bfe7dd94352abcaf34a200d00bc (DEFAULT_TABLE table=default.Table_85297cf02a6d4a06a5931f0203962992 [id=3289908651fc4a63ba05b7b30c2ed718]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.963433 23785 raft_consensus.cc:2243] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.963518 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f766bfe7dd94352abcaf34a200d00bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.963665 23785 raft_consensus.cc:2272] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.964098 23790 ts_tablet_manager.cc:1434] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251023 20:15:15.965178 23787 consensus_queue.cc:260] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.965628 23785 ts_tablet_manager.cc:1916] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.966353 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet e44dfb4fd3d04a58b5fcc6102e055ff1
I20251023 20:15:15.966782 23142 catalog_manager.cc:6719] ReplaceTablet: tablet e44dfb4fd3d04a58b5fcc6102e055ff1 deleted and replaced by tablet e5368813164949ff80e27bf59dcc931a
I20251023 20:15:15.966972 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 with delete_type TABLET_DATA_DELETED (replaced by tablet e5368813164949ff80e27bf59dcc931a) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.967305 23378 tablet_service.cc:1505] Processing CreateTablet for tablet c95db7b5b9fd46acae8ffa7119915907 (DEFAULT_TABLE table=default.Table_91596859556249fbb6e82b7dc5756a92 [id=5dff7019113a4323ae78b11f6b547b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.967525 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 with delete_type TABLET_DATA_DELETED (replaced by tablet e5368813164949ff80e27bf59dcc931a) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.967697 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95db7b5b9fd46acae8ffa7119915907. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:15.968633 23641 tablet_service.cc:1505] Processing CreateTablet for tablet c95db7b5b9fd46acae8ffa7119915907 (DEFAULT_TABLE table=default.Table_91596859556249fbb6e82b7dc5756a92 [id=5dff7019113a4323ae78b11f6b547b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:15.969020 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95db7b5b9fd46acae8ffa7119915907. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:15.969915 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e44dfb4fd3d04a58b5fcc6102e055ff1 already in progress: creating tablet
I20251023 20:15:15.971791 23785 ts_tablet_manager.cc:1929] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.973063 23784 tablet_replica.cc:333] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:15.974514 23787 ts_tablet_manager.cc:1434] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20251023 20:15:15.975620 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 64f515399c494aadbed2bd0eac171a77
I20251023 20:15:15.976015 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 50c75cce4b9d4c48800b2d0908676a14
I20251023 20:15:15.976101 23140 catalog_manager.cc:6719] ReplaceTablet: tablet 64f515399c494aadbed2bd0eac171a77 deleted and replaced by tablet ef1229b9e0384a82811f244d2bf4c8ce
I20251023 20:15:15.977677 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f515399c494aadbed2bd0eac171a77 with delete_type TABLET_DATA_DELETED (replaced by tablet ef1229b9e0384a82811f244d2bf4c8ce) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.977948 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.978369 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet dd456675077a4c73933c26e9a0313848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:15.978958 23783 tablet_replica.cc:333] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:15.979333 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 64f515399c494aadbed2bd0eac171a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f515399c494aadbed2bd0eac171a77 already in progress: creating tablet
W20251023 20:15:15.979583 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 50c75cce4b9d4c48800b2d0908676a14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50c75cce4b9d4c48800b2d0908676a14 already in progress: creating tablet
I20251023 20:15:15.973237 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 with delete_type TABLET_DATA_DELETED (replaced by tablet e5368813164949ff80e27bf59dcc931a) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.977020 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f515399c494aadbed2bd0eac171a77 with delete_type TABLET_DATA_DELETED (replaced by tablet ef1229b9e0384a82811f244d2bf4c8ce) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.977197 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.979948 23783 raft_consensus.cc:2243] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.980082 23784 raft_consensus.cc:2243] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:15.980118 23783 raft_consensus.cc:2272] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.980232 23784 raft_consensus.cc:2272] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:15.980921 23784 ts_tablet_manager.cc:1916] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.981040 23783 ts_tablet_manager.cc:1916] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:15.983381 23786 tablet_bootstrap.cc:492] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:15.983558 23786 ts_tablet_manager.cc:1403] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.000s
I20251023 20:15:15.984320 23786 raft_consensus.cc:359] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.984536 23786 raft_consensus.cc:385] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:15.984612 23786 raft_consensus.cc:740] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:15.984884 23786 consensus_queue.cc:260] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:15.985838 23786 ts_tablet_manager.cc:1434] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:15.987063 23783 ts_tablet_manager.cc:1929] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:15.988494 23783 log.cc:1199] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/dd456675077a4c73933c26e9a0313848
I20251023 20:15:15.989378 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_91596859556249fbb6e82b7dc5756a92" } modify_external_catalogs: true
I20251023 20:15:15.989562 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_91596859556249fbb6e82b7dc5756a92" } modify_external_catalogs: true
I20251023 20:15:15.990329 23783 ts_tablet_manager.cc:1950] T dd456675077a4c73933c26e9a0313848 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:15.990935 23786 tablet_bootstrap.cc:492] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:15.992633 23786 tablet_bootstrap.cc:654] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.992655 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4f134fc02d1c4a53a726595803ef505c" } modify_external_catalogs: true
I20251023 20:15:15.988139 23787 tablet_bootstrap.cc:492] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:15.992841 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4f134fc02d1c4a53a726595803ef505c" } modify_external_catalogs: true
I20251023 20:15:15.993677 23785 log.cc:1199] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/2177416169ca4d0d8368b1cac90c513e
I20251023 20:15:15.973886 23790 tablet_bootstrap.cc:492] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:15.986420 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.995131 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet e87648998c0b4297ab644d2c39ab2132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:15.988299 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f515399c494aadbed2bd0eac171a77 with delete_type TABLET_DATA_DELETED (replaced by tablet ef1229b9e0384a82811f244d2bf4c8ce) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:15.995951 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet dd456675077a4c73933c26e9a0313848 (table default.Table_da0691e8e6224cfebf17fe7df75954dd [id=a9e1acb22d6f423f924ad7417f9a986e]) successfully deleted
I20251023 20:15:15.994423 23787 tablet_bootstrap.cc:654] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.996634 23785 ts_tablet_manager.cc:1950] T 2177416169ca4d0d8368b1cac90c513e P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:15.998011 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 2177416169ca4d0d8368b1cac90c513e (table default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3 [id=2d40a1f71ab847df9127c80fd25f05af]) successfully deleted
I20251023 20:15:15.996949 23790 tablet_bootstrap.cc:654] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:15.998636 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 7f766bfe7dd94352abcaf34a200d00bc (DEFAULT_TABLE table=default.Table_85297cf02a6d4a06a5931f0203962992 [id=3289908651fc4a63ba05b7b30c2ed718]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.000973 23785 tablet_replica.cc:333] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.001658 23785 raft_consensus.cc:2243] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.002199 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f766bfe7dd94352abcaf34a200d00bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.002242 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_93ffb17658084632a48221ae255f149d" } modify_external_catalogs: true
I20251023 20:15:16.002358 23785 raft_consensus.cc:2272] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.002424 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_93ffb17658084632a48221ae255f149d" } modify_external_catalogs: true
I20251023 20:15:16.002482 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet c95db7b5b9fd46acae8ffa7119915907
I20251023 20:15:16.003708 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet c95db7b5b9fd46acae8ffa7119915907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.003906 23785 ts_tablet_manager.cc:1916] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.005156 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet c95db7b5b9fd46acae8ffa7119915907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.005241 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 2e338b982a564ea5b1b15c2146a96fa2
I20251023 20:15:16.005829 23784 ts_tablet_manager.cc:1929] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.005955 23784 log.cc:1199] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/536216374bae4bc481d78fabbed3567f
I20251023 20:15:16.006820 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:16.007629 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2e338b982a564ea5b1b15c2146a96fa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e338b982a564ea5b1b15c2146a96fa2 already in progress: creating tablet
I20251023 20:15:16.003413 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet c95db7b5b9fd46acae8ffa7119915907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.004391 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe (DEFAULT_TABLE table=default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.008677 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.008698 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_561d6afe24b246c9909547a248a5fe7f" } modify_external_catalogs: true
I20251023 20:15:16.008862 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_561d6afe24b246c9909547a248a5fe7f" } modify_external_catalogs: true
I20251023 20:15:16.008934 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 451955aee3aa4bcb82252bcefbbcaefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.012666 23787 tablet_bootstrap.cc:492] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.012991 23790 tablet_bootstrap.cc:492] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.013005 23787 ts_tablet_manager.cc:1403] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251023 20:15:16.013180 23790 ts_tablet_manager.cc:1403] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.000s
I20251023 20:15:16.013821 23787 raft_consensus.cc:359] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.014040 23787 raft_consensus.cc:385] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.013955 23790 raft_consensus.cc:359] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.014137 23787 raft_consensus.cc:740] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.014149 23790 raft_consensus.cc:385] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.007300 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.014367 23787 consensus_queue.cc:260] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.014798 23784 ts_tablet_manager.cc:1950] T 536216374bae4bc481d78fabbed3567f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.015385 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d05145bbb50343f38166de6b35bad890
I20251023 20:15:16.015738 23784 tablet_replica.cc:333] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.015918 23784 raft_consensus.cc:2243] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.016110 23784 raft_consensus.cc:2272] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.016090 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 536216374bae4bc481d78fabbed3567f (table default.Table_50a809e112244d0984bf222990b09883 [id=5991f18468ca47ac82800d008795e0dc]) successfully deleted
I20251023 20:15:16.016366 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet d05145bbb50343f38166de6b35bad890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.016597 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet d05145bbb50343f38166de6b35bad890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.014220 23790 raft_consensus.cc:740] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.016862 23783 tablet_replica.cc:333] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.017118 23783 raft_consensus.cc:2243] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.017057 23790 consensus_queue.cc:260] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.017633 23790 ts_tablet_manager.cc:1434] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251023 20:15:16.018523 23786 tablet_bootstrap.cc:492] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.018657 23786 ts_tablet_manager.cc:1403] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20251023 20:15:16.019227 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet d9b25166b00842948980cdfcacdd5df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.019505 23783 raft_consensus.cc:2272] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.019434 23786 raft_consensus.cc:359] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.019613 23786 raft_consensus.cc:385] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.019747 23786 raft_consensus.cc:740] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.020017 23786 consensus_queue.cc:260] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.020241 23783 ts_tablet_manager.cc:1916] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.020833 23786 ts_tablet_manager.cc:1434] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:16.021175 23786 tablet_bootstrap.cc:492] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.021550 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet d05145bbb50343f38166de6b35bad890 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.022693 23784 ts_tablet_manager.cc:1916] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.023041 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 487d9ee765d14aa2aa5668dc48fffe20
I20251023 20:15:16.024031 23790 tablet_bootstrap.cc:492] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.025564 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.025907 23786 tablet_bootstrap.cc:654] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.027098 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.027765 23790 tablet_bootstrap.cc:654] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.029902 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.030475 23787 ts_tablet_manager.cc:1434] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.017s user 0.000s sys 0.001s
I20251023 20:15:16.030474 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_5dbd1e92b4384903b369416019cf70b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.031087 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dbd1e92b4384903b369416019cf70b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.031955 23787 tablet_bootstrap.cc:492] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.033767 23787 tablet_bootstrap.cc:654] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.034569 23785 ts_tablet_manager.cc:1929] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.035076 23785 log.cc:1199] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/b125422cf17145d5aef418817082eb32
I20251023 20:15:16.035549 23785 ts_tablet_manager.cc:1950] T b125422cf17145d5aef418817082eb32 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.036094 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe (DEFAULT_TABLE table=default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.036499 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 451955aee3aa4bcb82252bcefbbcaefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.036875 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 1bcdec7c16c943079e20bb486aecb431 (DEFAULT_TABLE table=default.Table_6f178deaeeb94c4da6a40568c635e0b5 [id=cc451577eb8b4ba7937b8dbe4e859bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.036866 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 1bcdec7c16c943079e20bb486aecb431 (DEFAULT_TABLE table=default.Table_6f178deaeeb94c4da6a40568c635e0b5 [id=cc451577eb8b4ba7937b8dbe4e859bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.037283 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bcdec7c16c943079e20bb486aecb431. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.037282 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bcdec7c16c943079e20bb486aecb431. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.044247 23790 tablet_bootstrap.cc:492] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.044505 23790 ts_tablet_manager.cc:1403] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20251023 20:15:16.045168 23505 tablet_service.cc:1505] Processing CreateTablet for tablet c95db7b5b9fd46acae8ffa7119915907 (DEFAULT_TABLE table=default.Table_91596859556249fbb6e82b7dc5756a92 [id=5dff7019113a4323ae78b11f6b547b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.044678 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe (DEFAULT_TABLE table=default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.045878 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet b125422cf17145d5aef418817082eb32 (table default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:16.046218 23790 raft_consensus.cc:359] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.046584 23790 raft_consensus.cc:385] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.046773 23790 raft_consensus.cc:740] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.047056 23790 consensus_queue.cc:260] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.047448 23786 tablet_bootstrap.cc:492] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.047607 23786 ts_tablet_manager.cc:1403] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251023 20:15:16.048416 23786 raft_consensus.cc:359] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.048586 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 451955aee3aa4bcb82252bcefbbcaefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.048628 23786 raft_consensus.cc:385] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.048692 23786 raft_consensus.cc:740] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.049893 23790 ts_tablet_manager.cc:1434] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:16.050374 23786 consensus_queue.cc:260] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.051009 23786 ts_tablet_manager.cc:1434] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:16.051316 23786 tablet_bootstrap.cc:492] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.052687 23785 tablet_replica.cc:333] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.052965 23785 raft_consensus.cc:2243] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.053244 23785 raft_consensus.cc:2272] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.055354 23790 tablet_bootstrap.cc:492] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.057235 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.057322 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet e87648998c0b4297ab644d2c39ab2132 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.057771 23784 ts_tablet_manager.cc:1929] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.057834 23785 ts_tablet_manager.cc:1916] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.057998 23784 log.cc:1199] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/222ee7b9fa0a4bf69bb2ce8dab366f19
I20251023 20:15:16.058866 23784 ts_tablet_manager.cc:1950] T 222ee7b9fa0a4bf69bb2ce8dab366f19 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.059878 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 222ee7b9fa0a4bf69bb2ce8dab366f19 (table default.Table_b6760837eeee4b5082522107b87d72f5 [id=6e38fec2eebd48b8a6364be16ec99997]) successfully deleted
I20251023 20:15:16.060667 23790 tablet_bootstrap.cc:654] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.061863 23784 tablet_replica.cc:333] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.062124 23784 raft_consensus.cc:2243] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.062348 23784 raft_consensus.cc:2272] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.063113 23784 ts_tablet_manager.cc:1916] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.063714 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7bfeb621c43543089ded2b36149f7ced"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.064188 23790 tablet_bootstrap.cc:492] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:16.064364 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bfeb621c43543089ded2b36149f7ced in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.064409 23790 ts_tablet_manager.cc:1403] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20251023 20:15:16.065357 23790 raft_consensus.cc:359] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.065670 23790 raft_consensus.cc:385] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.065789 23790 raft_consensus.cc:740] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.066164 23790 consensus_queue.cc:260] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.068316 23790 ts_tablet_manager.cc:1434] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:16.069517 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 56fd87a212ff4ce893a44597ec038be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.069980 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 1bcdec7c16c943079e20bb486aecb431 (DEFAULT_TABLE table=default.Table_6f178deaeeb94c4da6a40568c635e0b5 [id=cc451577eb8b4ba7937b8dbe4e859bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.070981 23370 tablet_service.cc:1505] Processing CreateTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 (DEFAULT_TABLE table=default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.071090 23507 tablet_service.cc:1505] Processing CreateTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 (DEFAULT_TABLE table=default.Table_4d358925f1254129a8b42bfca11f8870 [id=a1701c81a217439197ef53ff39675ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.071218 23504 tablet_service.cc:1505] Processing CreateTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 (DEFAULT_TABLE table=default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.071471 23367 tablet_service.cc:1505] Processing CreateTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 (DEFAULT_TABLE table=default.Table_4d358925f1254129a8b42bfca11f8870 [id=a1701c81a217439197ef53ff39675ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.073071 23645 tablet_service.cc:1505] Processing CreateTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 (DEFAULT_TABLE table=default.Table_4d358925f1254129a8b42bfca11f8870 [id=a1701c81a217439197ef53ff39675ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.073067 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 8e19f64a9ac44efba92928d0fa16819a (DEFAULT_TABLE table=default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.073302 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95db7b5b9fd46acae8ffa7119915907. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.073663 23637 tablet_service.cc:1505] Processing CreateTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 (DEFAULT_TABLE table=default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.074173 23379 tablet_service.cc:1505] Processing CreateTablet for tablet 8e19f64a9ac44efba92928d0fa16819a (DEFAULT_TABLE table=default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.074678 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31aebfc87734ffeb2c0b25c07cb3f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.075883 23786 tablet_bootstrap.cc:654] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.077201 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e61de7a93d450d9ed79eef82d7e5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.077611 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e19f64a9ac44efba92928d0fa16819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.081766 23790 tablet_bootstrap.cc:492] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.085381 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31aebfc87734ffeb2c0b25c07cb3f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.085646 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e19f64a9ac44efba92928d0fa16819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.089696 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e61de7a93d450d9ed79eef82d7e5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.089807 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_cbcc7255010047c288af86f302dfd125" } new_table_name: "default.Table_710f1f5465e64779af50f553912b324d" modify_external_catalogs: true
W20251023 20:15:16.096936 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 040a44e272b7410c98c109fbceb19192 consensus state has no leader
I20251023 20:15:16.100524 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bcdec7c16c943079e20bb486aecb431. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.100562 23783 ts_tablet_manager.cc:1929] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.100694 23783 log.cc:1199] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d05145bbb50343f38166de6b35bad890
I20251023 20:15:16.100862 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0e61de7a93d450d9ed79eef82d7e5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.101122 23783 ts_tablet_manager.cc:1950] T d05145bbb50343f38166de6b35bad890 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:16.101258 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31aebfc87734ffeb2c0b25c07cb3f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.102072 23783 tablet_replica.cc:333] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.102260 23783 raft_consensus.cc:2243] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.102442 23783 raft_consensus.cc:2272] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.103415 23787 tablet_bootstrap.cc:492] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.103662 23787 ts_tablet_manager.cc:1403] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20251023 20:15:16.105194 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d05145bbb50343f38166de6b35bad890 (table default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]) successfully deleted
I20251023 20:15:16.106372 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 6558265ff9e249c68123577958f1401a (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.106462 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 949e31d4a9ee4579843ed54153e22c7d from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.106806 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6558265ff9e249c68123577958f1401a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.104529 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 8e19f64a9ac44efba92928d0fa16819a (DEFAULT_TABLE table=default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.110229 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e19f64a9ac44efba92928d0fa16819a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.106698 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 585a9af140d9479fb83d13e68a62300c (DEFAULT_TABLE table=default.Table_c3d2d37dd2bd414fae4c968fc3f52a25 [id=97396cb497da42f6af247dd68249182e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.113229 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 585a9af140d9479fb83d13e68a62300c (DEFAULT_TABLE table=default.Table_c3d2d37dd2bd414fae4c968fc3f52a25 [id=97396cb497da42f6af247dd68249182e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.113497 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585a9af140d9479fb83d13e68a62300c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.113868 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585a9af140d9479fb83d13e68a62300c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.107077 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 6558265ff9e249c68123577958f1401a (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.117519 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6558265ff9e249c68123577958f1401a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.118933 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 949e31d4a9ee4579843ed54153e22c7d
I20251023 20:15:16.119387 23140 catalog_manager.cc:6719] ReplaceTablet: tablet 949e31d4a9ee4579843ed54153e22c7d deleted and replaced by tablet 9c0aeac2f20c4ca48c130a0c851f290f
I20251023 20:15:16.119807 23783 ts_tablet_manager.cc:1916] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.125559 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 949e31d4a9ee4579843ed54153e22c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 9c0aeac2f20c4ca48c130a0c851f290f) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.125990 23783 ts_tablet_manager.cc:1929] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.126407 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 949e31d4a9ee4579843ed54153e22c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 9c0aeac2f20c4ca48c130a0c851f290f) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.126688 23783 log.cc:1199] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d9b25166b00842948980cdfcacdd5df2
I20251023 20:15:16.127082 23783 ts_tablet_manager.cc:1950] T d9b25166b00842948980cdfcacdd5df2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:16.128183 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d9b25166b00842948980cdfcacdd5df2 (table default.Table_f66ae376b73c46d180bde83fb80dc7bb [id=4f40834f1fae488bb070b0c40c191ee8]) successfully deleted
W20251023 20:15:16.128846 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c95db7b5b9fd46acae8ffa7119915907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c95db7b5b9fd46acae8ffa7119915907 already in progress: creating tablet
W20251023 20:15:16.129477 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 487d9ee765d14aa2aa5668dc48fffe20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 487d9ee765d14aa2aa5668dc48fffe20 already in progress: creating tablet
I20251023 20:15:16.130235 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 585a9af140d9479fb83d13e68a62300c (DEFAULT_TABLE table=default.Table_c3d2d37dd2bd414fae4c968fc3f52a25 [id=97396cb497da42f6af247dd68249182e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.130265 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 6558265ff9e249c68123577958f1401a (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.130764 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585a9af140d9479fb83d13e68a62300c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.130959 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6558265ff9e249c68123577958f1401a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.132459 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 949e31d4a9ee4579843ed54153e22c7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 949e31d4a9ee4579843ed54153e22c7d already in progress: creating tablet
I20251023 20:15:16.133152 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 949e31d4a9ee4579843ed54153e22c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 9c0aeac2f20c4ca48c130a0c851f290f) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.134296 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0d5856c857fb40e68632458d791c47ba" } new_table_name: "default.Table_0197c34b4744436b8ba95be1ba33be9f" modify_external_catalogs: true
I20251023 20:15:16.134439 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7bfeb621c43543089ded2b36149f7ced" } new_table_name: "default.Table_67a76ff63d4d411582ab8e10ac4f6924" modify_external_catalogs: true
I20251023 20:15:16.134649 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_344bdb8a8edd4641b4e9786c5d6ce520" } new_table_name: "default.Table_4fba73266b6d4d36958e8f0566eb624e" modify_external_catalogs: true
I20251023 20:15:16.134691 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e8cec02c09a44acab531a2f71ebb0a73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.135697 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_85297cf02a6d4a06a5931f0203962992" } new_table_name: "default.Table_f2f55454e9b04d77860924e828522077" modify_external_catalogs: true
W20251023 20:15:16.135218 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8cec02c09a44acab531a2f71ebb0a73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.138767 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_94257ba5c56c4269842dcd5dd0d19461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.139256 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94257ba5c56c4269842dcd5dd0d19461 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.140435 23787 raft_consensus.cc:359] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.140492 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_2ca7b39d3f2341fb88826d33392f1d13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.140801 23787 raft_consensus.cc:385] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.140921 23787 raft_consensus.cc:740] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:16.141116 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ca7b39d3f2341fb88826d33392f1d13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:16.143155 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3894dd4feeea4c16861a6c9520059474 has no consensus state
W20251023 20:15:16.144052 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc8f615c621043da88d52f3964b0a0ec consensus state has no leader
I20251023 20:15:16.145937 23787 consensus_queue.cc:260] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.146986 23787 ts_tablet_manager.cc:1434] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251023 20:15:16.147395 23787 tablet_bootstrap.cc:492] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.148156 23790 tablet_bootstrap.cc:654] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.149125 23785 ts_tablet_manager.cc:1929] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.151583 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f205f4e6d547229f3026c16a6fd485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.151819 23785 log.cc:1199] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/6ae1876fa334495ca95dc2e9e6b4a0fd
I20251023 20:15:16.151888 23783 tablet_replica.cc:333] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.152076 23783 raft_consensus.cc:2243] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.152154 23786 tablet_bootstrap.cc:492] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.152192 23785 ts_tablet_manager.cc:1950] T 6ae1876fa334495ca95dc2e9e6b4a0fd P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.152313 23786 ts_tablet_manager.cc:1403] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.004s
I20251023 20:15:16.152311 23783 raft_consensus.cc:2272] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:16.152647 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f766bfe7dd94352abcaf34a200d00bc consensus state has no leader
I20251023 20:15:16.153223 23783 ts_tablet_manager.cc:1916] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.153141 23786 raft_consensus.cc:359] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.153323 23786 raft_consensus.cc:385] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.153430 23786 raft_consensus.cc:740] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.153666 23786 consensus_queue.cc:260] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.154193 23786 ts_tablet_manager.cc:1434] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20251023 20:15:16.154414 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4bb6abe3ef848e2831839de6a28eb0f consensus state has no leader
I20251023 20:15:16.155256 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 6ae1876fa334495ca95dc2e9e6b4a0fd (table default.Table_8b24573ba4864a13a123e5d8d2bf9448 [id=fb7af3cee68a4f3c9307a5876db279c1]) successfully deleted
W20251023 20:15:16.155536 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e44dfb4fd3d04a58b5fcc6102e055ff1 already in progress: creating tablet
W20251023 20:15:16.156163 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 50c75cce4b9d4c48800b2d0908676a14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50c75cce4b9d4c48800b2d0908676a14 already in progress: creating tablet
W20251023 20:15:16.156431 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 64f515399c494aadbed2bd0eac171a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f515399c494aadbed2bd0eac171a77 already in progress: creating tablet
I20251023 20:15:16.156873 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d6c2066db27a45ccbbd39d93a44c5833"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.157335 23784 ts_tablet_manager.cc:1929] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.157477 23784 log.cc:1199] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/b09c0412f1eb48df8de542ca687a7b48
W20251023 20:15:16.157454 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6c2066db27a45ccbbd39d93a44c5833 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.157814 23784 ts_tablet_manager.cc:1950] T b09c0412f1eb48df8de542ca687a7b48 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.158630 23785 tablet_replica.cc:333] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.158685 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_220abf897e5a4a07bd1af3ab929cb2d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.158828 23785 raft_consensus.cc:2243] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.159005 23785 raft_consensus.cc:2272] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:16.159162 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_220abf897e5a4a07bd1af3ab929cb2d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.159766 23785 ts_tablet_manager.cc:1916] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.160226 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6d625b139835401587825737c53000e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.160550 23783 ts_tablet_manager.cc:1929] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:16.160672 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d625b139835401587825737c53000e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.160708 23783 log.cc:1199] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/73f205f4e6d547229f3026c16a6fd485
I20251023 20:15:16.161173 23783 ts_tablet_manager.cc:1950] T 73f205f4e6d547229f3026c16a6fd485 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:16.162524 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 73f205f4e6d547229f3026c16a6fd485 (table default.Table_43bf18200b234de68377af27f882f395 [id=da87b3d063a64aab92f939df02727144]) successfully deleted
I20251023 20:15:16.163434 23786 tablet_bootstrap.cc:492] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:16.163893 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c95db7b5b9fd46acae8ffa7119915907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c95db7b5b9fd46acae8ffa7119915907 already in progress: creating tablet
W20251023 20:15:16.164160 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2e338b982a564ea5b1b15c2146a96fa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e338b982a564ea5b1b15c2146a96fa2 already in progress: creating tablet
I20251023 20:15:16.165072 23790 tablet_bootstrap.cc:492] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.165288 23790 ts_tablet_manager.cc:1403] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.000s
I20251023 20:15:16.166190 23790 raft_consensus.cc:359] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.166450 23790 raft_consensus.cc:385] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.166793 23790 raft_consensus.cc:740] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.166887 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet 487d9ee765d14aa2aa5668dc48fffe20 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.167114 23790 consensus_queue.cc:260] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.171943 23790 ts_tablet_manager.cc:1434] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:16.173023 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet dd456675077a4c73933c26e9a0313848 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.173856 23790 tablet_bootstrap.cc:492] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.175680 23790 tablet_bootstrap.cc:654] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.180393 23785 ts_tablet_manager.cc:1929] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.181416 23785 log.cc:1199] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d05145bbb50343f38166de6b35bad890
I20251023 20:15:16.181898 23785 ts_tablet_manager.cc:1950] T d05145bbb50343f38166de6b35bad890 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.182178 23786 tablet_bootstrap.cc:654] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.183432 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet 955412de8bd1482ea7304872b48f38c0 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.183816 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet 0d73da547fd04557abd5aba73e76dd18 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.186444 23785 tablet_replica.cc:333] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.186655 23785 raft_consensus.cc:2243] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.186890 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d05145bbb50343f38166de6b35bad890 (table default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]) successfully deleted
I20251023 20:15:16.187022 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_3fd19e6c8ca54391bd33c1c235d72938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.187393 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 487d9ee765d14aa2aa5668dc48fffe20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 487d9ee765d14aa2aa5668dc48fffe20 already in progress: creating tablet
I20251023 20:15:16.187959 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ea61082402e14334acc5162a368129e3" } modify_external_catalogs: true
I20251023 20:15:16.188134 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ea61082402e14334acc5162a368129e3" } modify_external_catalogs: true
I20251023 20:15:16.188198 23785 raft_consensus.cc:2272] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.189430 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e186a061f84042cd82645b37beae1da1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.189554 23785 ts_tablet_manager.cc:1916] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.193248 23794 raft_consensus.cc:493] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:16.193444 23794 raft_consensus.cc:515] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.194134 23794 leader_election.cc:290] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:16.194346 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.194679 23784 tablet_replica.cc:333] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.195330 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet b09c0412f1eb48df8de542ca687a7b48 (table default.Table_45640565c4a24ff88274bc4e62b865ff [id=18ee017a8d484c7b9ff7dbbceaa5f7ac]) successfully deleted
W20251023 20:15:16.195603 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e44dfb4fd3d04a58b5fcc6102e055ff1 already in progress: creating tablet
W20251023 20:15:16.195860 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 64f515399c494aadbed2bd0eac171a77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64f515399c494aadbed2bd0eac171a77 already in progress: creating tablet
W20251023 20:15:16.196051 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 50c75cce4b9d4c48800b2d0908676a14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50c75cce4b9d4c48800b2d0908676a14 already in progress: creating tablet
I20251023 20:15:16.196560 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ad21e3888a4915ae5f252c2fca64cb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:16.196650 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ad21e3888a4915ae5f252c2fca64cb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:16.196895 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e186a061f84042cd82645b37beae1da1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.197160 23787 tablet_bootstrap.cc:654] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:16.197191 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fd19e6c8ca54391bd33c1c235d72938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.198971 23784 raft_consensus.cc:2243] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.199438 23784 raft_consensus.cc:2272] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.200335 23784 ts_tablet_manager.cc:1916] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:16.198431 23470 leader_election.cc:343] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:16.203055 23471 leader_election.cc:343] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:16.203497 23471 leader_election.cc:304] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:16.204185 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.204542 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_67a7910474894d659750844c77625a9d" } modify_external_catalogs: true
I20251023 20:15:16.204725 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_67a7910474894d659750844c77625a9d" } modify_external_catalogs: true
I20251023 20:15:16.204792 23786 tablet_bootstrap.cc:492] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.204955 23786 ts_tablet_manager.cc:1403] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.004s
I20251023 20:15:16.205497 23785 ts_tablet_manager.cc:1929] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.205653 23785 log.cc:1199] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e87648998c0b4297ab644d2c39ab2132
I20251023 20:15:16.205802 23786 raft_consensus.cc:359] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.205992 23785 ts_tablet_manager.cc:1950] T e87648998c0b4297ab644d2c39ab2132 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.205997 23786 raft_consensus.cc:385] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.206071 23786 raft_consensus.cc:740] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.205971 23794 raft_consensus.cc:2749] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:16.206364 23786 consensus_queue.cc:260] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.206846 23786 ts_tablet_manager.cc:1434] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:16.206908 23785 tablet_replica.cc:333] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.207062 23785 raft_consensus.cc:2243] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.207144 23786 tablet_bootstrap.cc:492] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.207331 23785 raft_consensus.cc:2272] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.208755 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet e87648998c0b4297ab644d2c39ab2132 (table default.Table_57c800e050ac4e75a5cdbfff336ec344 [id=823681dedfc3477d995ad3695c4a9541]) successfully deleted
I20251023 20:15:16.210057 23786 tablet_bootstrap.cc:654] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.210943 23787 tablet_bootstrap.cc:492] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.211169 23787 ts_tablet_manager.cc:1403] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.002s
I20251023 20:15:16.212260 23787 raft_consensus.cc:359] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.212525 23787 raft_consensus.cc:385] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.212687 23787 raft_consensus.cc:740] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.213001 23787 consensus_queue.cc:260] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.215436 23787 ts_tablet_manager.cc:1434] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:16.216676 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d73da547fd04557abd5aba73e76dd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.217844 23787 tablet_bootstrap.cc:492] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.220901 23784 ts_tablet_manager.cc:1929] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.221101 23784 log.cc:1199] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e87648998c0b4297ab644d2c39ab2132
I20251023 20:15:16.221515 23784 ts_tablet_manager.cc:1950] T e87648998c0b4297ab644d2c39ab2132 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.222199 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a2d1315afa0468c956adf64e1bd9cf0 with delete_type TABLET_DATA_DELETED (replaced by tablet 24528de9d0a84eddb091dec4f1d2b4e2) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.222496 23783 tablet_replica.cc:333] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.222668 23783 raft_consensus.cc:2243] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.222744 23785 ts_tablet_manager.cc:1916] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.222817 23783 raft_consensus.cc:2272] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.223371 23783 ts_tablet_manager.cc:1916] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.224673 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b6760837eeee4b5082522107b87d72f5" } modify_external_catalogs: true
I20251023 20:15:16.224862 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b6760837eeee4b5082522107b87d72f5" } modify_external_catalogs: true
I20251023 20:15:16.226723 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_265f006bfcc2450f964410c799f035f9" } modify_external_catalogs: true
I20251023 20:15:16.226894 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_265f006bfcc2450f964410c799f035f9" } modify_external_catalogs: true
I20251023 20:15:16.228940 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet e87648998c0b4297ab644d2c39ab2132 (table default.Table_57c800e050ac4e75a5cdbfff336ec344 [id=823681dedfc3477d995ad3695c4a9541]) successfully deleted
I20251023 20:15:16.229126 23784 tablet_replica.cc:333] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.229323 23784 raft_consensus.cc:2243] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.229552 23783 ts_tablet_manager.cc:1929] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:16.229578 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c95db7b5b9fd46acae8ffa7119915907 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c95db7b5b9fd46acae8ffa7119915907 already in progress: creating tablet
I20251023 20:15:16.229779 23783 log.cc:1199] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/9a2d1315afa0468c956adf64e1bd9cf0
W20251023 20:15:16.229836 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2e338b982a564ea5b1b15c2146a96fa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e338b982a564ea5b1b15c2146a96fa2 already in progress: creating tablet
I20251023 20:15:16.231205 23784 raft_consensus.cc:2272] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.231462 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_82f6a78855d1488b9a8085c3bfbebab3" } modify_external_catalogs: true
I20251023 20:15:16.231648 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_82f6a78855d1488b9a8085c3bfbebab3" } modify_external_catalogs: true
I20251023 20:15:16.231487 23785 ts_tablet_manager.cc:1929] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.231937 23785 log.cc:1199] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/56fd87a212ff4ce893a44597ec038be9
I20251023 20:15:16.232215 23784 ts_tablet_manager.cc:1916] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.234055 23785 ts_tablet_manager.cc:1950] T 56fd87a212ff4ce893a44597ec038be9 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.235527 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 3894dd4feeea4c16861a6c9520059474 (DEFAULT_TABLE table=default.Table_67a76ff63d4d411582ab8e10ac4f6924 [id=fb6dc9a69da44f35a0184490d90a1e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.235822 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 3894dd4feeea4c16861a6c9520059474 (DEFAULT_TABLE table=default.Table_67a76ff63d4d411582ab8e10ac4f6924 [id=fb6dc9a69da44f35a0184490d90a1e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.235960 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3894dd4feeea4c16861a6c9520059474. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.236205 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3894dd4feeea4c16861a6c9520059474. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.236557 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 56fd87a212ff4ce893a44597ec038be9 (table default.Table_0acacb2edf1c461baf55cc00e9a69ff6 [id=8478b83c0ee645fba6c41151e56246b7]) successfully deleted
I20251023 20:15:16.237037 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (DEFAULT_TABLE table=default.Table_5dbd1e92b4384903b369416019cf70b9 [id=84aabbb61e1c4d778b8e07ebdddbc477]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.237461 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.241415 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 4e72922aae98430da968874f44603317 (DEFAULT_TABLE table=default.Table_eb3f472232804ea98a49fe822c8e9118 [id=ad988b9ca869431fb987604850e97fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.241484 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 3894dd4feeea4c16861a6c9520059474 (DEFAULT_TABLE table=default.Table_67a76ff63d4d411582ab8e10ac4f6924 [id=fb6dc9a69da44f35a0184490d90a1e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.241828 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e72922aae98430da968874f44603317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.241888 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3894dd4feeea4c16861a6c9520059474. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.241883 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (DEFAULT_TABLE table=default.Table_5dbd1e92b4384903b369416019cf70b9 [id=84aabbb61e1c4d778b8e07ebdddbc477]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.242208 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.242324 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 4e72922aae98430da968874f44603317 (DEFAULT_TABLE table=default.Table_eb3f472232804ea98a49fe822c8e9118 [id=ad988b9ca869431fb987604850e97fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.242627 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e72922aae98430da968874f44603317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.245187 23794 raft_consensus.cc:493] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:16.245357 23794 raft_consensus.cc:515] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.245469 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (DEFAULT_TABLE table=default.Table_5dbd1e92b4384903b369416019cf70b9 [id=84aabbb61e1c4d778b8e07ebdddbc477]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.245816 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.246094 23794 leader_election.cc:290] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:16.246474 23641 tablet_service.cc:1505] Processing CreateTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 (DEFAULT_TABLE table=default.Table_0898b76943db42e98d75a6e39cb8fade [id=936e505ef2f7447db8f9cb85af99c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.246894 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f8d8184a68453abd2792ccc7740aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.248612 23783 ts_tablet_manager.cc:1950] T 9a2d1315afa0468c956adf64e1bd9cf0 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:16.248737 23504 tablet_service.cc:1505] Processing CreateTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 (DEFAULT_TABLE table=default.Table_0898b76943db42e98d75a6e39cb8fade [id=936e505ef2f7447db8f9cb85af99c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.249058 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f8d8184a68453abd2792ccc7740aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.249658 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 9a2d1315afa0468c956adf64e1bd9cf0 (table default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
I20251023 20:15:16.250100 23379 tablet_service.cc:1505] Processing CreateTablet for tablet 4e72922aae98430da968874f44603317 (DEFAULT_TABLE table=default.Table_eb3f472232804ea98a49fe822c8e9118 [id=ad988b9ca869431fb987604850e97fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.250494 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e72922aae98430da968874f44603317. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.252177 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0cd3fe0039de476c813272d09a9ba564"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.252730 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cd3fe0039de476c813272d09a9ba564 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.253463 23367 tablet_service.cc:1505] Processing CreateTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 (DEFAULT_TABLE table=default.Table_0898b76943db42e98d75a6e39cb8fade [id=936e505ef2f7447db8f9cb85af99c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.253883 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0f8d8184a68453abd2792ccc7740aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.254243 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d73da547fd04557abd5aba73e76dd18" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:16.254767 23787 tablet_bootstrap.cc:654] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.257405 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 83da09ee0d10435ba89a19dc378d34eb (DEFAULT_TABLE table=default.Table_bb6092ff59da4b9988c720c4987d7c35 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.257520 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 955412de8bd1482ea7304872b48f38c0
I20251023 20:15:16.257812 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da09ee0d10435ba89a19dc378d34eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.258031 23139 catalog_manager.cc:6719] ReplaceTablet: tablet 955412de8bd1482ea7304872b48f38c0 deleted and replaced by tablet 283166bbd8004ce3bca9f2ecbdbcbf48
I20251023 20:15:16.258392 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 2d698ffe7d154ef8b1994ffdd4b0c648
I20251023 20:15:16.259074 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 23225d31c2e647398aa58494c627507e
I20251023 20:15:16.260890 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.261214 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 955412de8bd1482ea7304872b48f38c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 283166bbd8004ce3bca9f2ecbdbcbf48) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.261576 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:16.262099 23470 leader_election.cc:343] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:16.262705 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 23225d31c2e647398aa58494c627507e (DEFAULT_TABLE table=default.Table_67a7910474894d659750844c77625a9d [id=82c65a189d13499c8471cb1ce0894c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.263110 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23225d31c2e647398aa58494c627507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.263160 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 90d58bceecba442898a3d742fa49a4a2 (DEFAULT_TABLE table=default.Table_6cf378e7dc9b4a37acaec754f2edc359 [id=e73714e917184f788bf57815006ec75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.263504 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d58bceecba442898a3d742fa49a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.263473 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 9fc7e18054c440098dc2470143ea7fae (DEFAULT_TABLE table=default.Table_265f006bfcc2450f964410c799f035f9 [id=f0872a169a604701b06c246f1af67ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.263840 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc7e18054c440098dc2470143ea7fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.264712 23790 tablet_bootstrap.cc:492] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.264878 23790 ts_tablet_manager.cc:1403] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.003s
I20251023 20:15:16.266088 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.266315 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 83da09ee0d10435ba89a19dc378d34eb (DEFAULT_TABLE table=default.Table_bb6092ff59da4b9988c720c4987d7c35 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.266459 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 90d58bceecba442898a3d742fa49a4a2 (DEFAULT_TABLE table=default.Table_6cf378e7dc9b4a37acaec754f2edc359 [id=e73714e917184f788bf57815006ec75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.266763 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da09ee0d10435ba89a19dc378d34eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.266932 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d58bceecba442898a3d742fa49a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.267943 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 23225d31c2e647398aa58494c627507e (DEFAULT_TABLE table=default.Table_67a7910474894d659750844c77625a9d [id=82c65a189d13499c8471cb1ce0894c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.268299 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23225d31c2e647398aa58494c627507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.269335 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.270072 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d73da547fd04557abd5aba73e76dd18" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:16.270395 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 955412de8bd1482ea7304872b48f38c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 283166bbd8004ce3bca9f2ecbdbcbf48) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.270455 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.270766 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.271282 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 90d58bceecba442898a3d742fa49a4a2 (DEFAULT_TABLE table=default.Table_6cf378e7dc9b4a37acaec754f2edc359 [id=e73714e917184f788bf57815006ec75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.271278 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 83da09ee0d10435ba89a19dc378d34eb (DEFAULT_TABLE table=default.Table_bb6092ff59da4b9988c720c4987d7c35 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.271689 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d58bceecba442898a3d742fa49a4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.273214 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da09ee0d10435ba89a19dc378d34eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.278204 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d8c675f452ec4b1ea9609eec6926317b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.278795 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8c675f452ec4b1ea9609eec6926317b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.271283 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 23225d31c2e647398aa58494c627507e (DEFAULT_TABLE table=default.Table_67a7910474894d659750844c77625a9d [id=82c65a189d13499c8471cb1ce0894c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.279709 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23225d31c2e647398aa58494c627507e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.279969 23790 raft_consensus.cc:359] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:16.280584 23471 leader_election.cc:343] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:16.280740 23471 leader_election.cc:304] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:16.281252 23794 raft_consensus.cc:2749] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:16.281986 23790 raft_consensus.cc:385] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.284636 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet dafb9db710f94c328186e3179bf0b319
I20251023 20:15:16.285738 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 5416da30c66645c19d36112980608527
I20251023 20:15:16.289481 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 5416da30c66645c19d36112980608527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.289690 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.290055 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 5416da30c66645c19d36112980608527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.292761 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 955412de8bd1482ea7304872b48f38c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 283166bbd8004ce3bca9f2ecbdbcbf48) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.282097 23790 raft_consensus.cc:740] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:16.294598 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d698ffe7d154ef8b1994ffdd4b0c648 already in progress: creating tablet
I20251023 20:15:16.295024 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:16.295171 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 955412de8bd1482ea7304872b48f38c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 955412de8bd1482ea7304872b48f38c0 already in progress: creating tablet
W20251023 20:15:16.295464 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 23225d31c2e647398aa58494c627507e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23225d31c2e647398aa58494c627507e already in progress: creating tablet
I20251023 20:15:16.294818 23790 consensus_queue.cc:260] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:16.295856 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet dafb9db710f94c328186e3179bf0b319 because the tablet was not found. No further retry: Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319
I20251023 20:15:16.295989 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:16.296160 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5416da30c66645c19d36112980608527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5416da30c66645c19d36112980608527 already in progress: creating tablet
I20251023 20:15:16.296204 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 5416da30c66645c19d36112980608527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.298161 23786 tablet_bootstrap.cc:492] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.298555 23786 ts_tablet_manager.cc:1403] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.010s
I20251023 20:15:16.299439 23786 raft_consensus.cc:359] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.299969 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 9fc7e18054c440098dc2470143ea7fae (DEFAULT_TABLE table=default.Table_265f006bfcc2450f964410c799f035f9 [id=f0872a169a604701b06c246f1af67ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.300173 23790 ts_tablet_manager.cc:1434] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20251023 20:15:16.300262 23786 raft_consensus.cc:385] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.300433 23786 raft_consensus.cc:740] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.300706 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 9fc7e18054c440098dc2470143ea7fae
I20251023 20:15:16.300740 23786 consensus_queue.cc:260] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.301280 23786 ts_tablet_manager.cc:1434] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:16.301568 23786 tablet_bootstrap.cc:492] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.302129 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.300412 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 9fc7e18054c440098dc2470143ea7fae (DEFAULT_TABLE table=default.Table_265f006bfcc2450f964410c799f035f9 [id=f0872a169a604701b06c246f1af67ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.304323 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.305678 23637 tablet_service.cc:1505] Processing CreateTablet for tablet e418a62d00824cff89b37a7bace5c469 (DEFAULT_TABLE table=default.Table_1662565661d643d8b4ff35b3e84f0351 [id=751ae1aba5564ffa8dd6274778d1b8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.306213 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e418a62d00824cff89b37a7bace5c469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.311065 23377 tablet_service.cc:1505] Processing CreateTablet for tablet e418a62d00824cff89b37a7bace5c469 (DEFAULT_TABLE table=default.Table_1662565661d643d8b4ff35b3e84f0351 [id=751ae1aba5564ffa8dd6274778d1b8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.312232 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_710f1f5465e64779af50f553912b324d" } new_table_name: "default.Table_07457cc5a3664344ae16b8a6fcb491af" modify_external_catalogs: true
W20251023 20:15:16.313628 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9fc7e18054c440098dc2470143ea7fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fc7e18054c440098dc2470143ea7fae already in progress: creating tablet
I20251023 20:15:16.315464 23143 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_363dbff3b2a046b190cdf7ce1a8572d6 [id=78fe983663c94eabab93ee902bde6998]). Expected version 3 got 0
I20251023 20:15:16.315460 23366 tablet_service.cc:1505] Processing CreateTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce (DEFAULT_TABLE table=default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.315649 23641 tablet_service.cc:1505] Processing CreateTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce (DEFAULT_TABLE table=default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.316159 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1229b9e0384a82811f244d2bf4c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.318169 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc7e18054c440098dc2470143ea7fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.303857 23786 tablet_bootstrap.cc:654] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.312659 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.314404 23501 tablet_service.cc:1505] Processing CreateTablet for tablet e418a62d00824cff89b37a7bace5c469 (DEFAULT_TABLE table=default.Table_1662565661d643d8b4ff35b3e84f0351 [id=751ae1aba5564ffa8dd6274778d1b8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.316893 23508 tablet_service.cc:1505] Processing CreateTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce (DEFAULT_TABLE table=default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.318766 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1229b9e0384a82811f244d2bf4c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.321722 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e418a62d00824cff89b37a7bace5c469. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.326074 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 949e31d4a9ee4579843ed54153e22c7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 949e31d4a9ee4579843ed54153e22c7d already in progress: creating tablet
I20251023 20:15:16.326125 23363 tablet_service.cc:1552] Processing DeleteTablet for tablet b09c0412f1eb48df8de542ca687a7b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:16.329236 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 040a44e272b7410c98c109fbceb19192 consensus state has no leader
I20251023 20:15:16.330545 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc7e18054c440098dc2470143ea7fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.332432 23641 tablet_service.cc:1505] Processing CreateTablet for tablet dafb9db710f94c328186e3179bf0b319 (DEFAULT_TABLE table=default.Table_82f6a78855d1488b9a8085c3bfbebab3 [id=f033b532f86d452c8f14e7f40ead8d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.332916 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafb9db710f94c328186e3179bf0b319. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.333583 23790 tablet_bootstrap.cc:492] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.335294 23790 tablet_bootstrap.cc:654] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.338661 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e418a62d00824cff89b37a7bace5c469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.339510 23785 tablet_replica.cc:333] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.339679 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1229b9e0384a82811f244d2bf4c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.339728 23785 raft_consensus.cc:2243] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.339893 23785 raft_consensus.cc:2272] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.340693 23785 ts_tablet_manager.cc:1916] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.342320 23784 ts_tablet_manager.cc:1929] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.342483 23784 log.cc:1199] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d05145bbb50343f38166de6b35bad890
I20251023 20:15:16.342821 23784 ts_tablet_manager.cc:1950] T d05145bbb50343f38166de6b35bad890 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.344023 23784 tablet_replica.cc:333] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.344216 23784 raft_consensus.cc:2243] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.344395 23784 raft_consensus.cc:2272] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.345472 23784 ts_tablet_manager.cc:1916] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.346771 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d6c2066db27a45ccbbd39d93a44c5833" } new_table_name: "default.Table_7bbb68c0efad486c9267e6330c620301" modify_external_catalogs: true
I20251023 20:15:16.348556 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d05145bbb50343f38166de6b35bad890 (table default.Table_93ffb17658084632a48221ae255f149d [id=99915ca206cd4f2aac23843e8bd6547e]) successfully deleted
W20251023 20:15:16.348817 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 487d9ee765d14aa2aa5668dc48fffe20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 487d9ee765d14aa2aa5668dc48fffe20 already in progress: creating tablet
I20251023 20:15:16.351472 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet 5416da30c66645c19d36112980608527 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.351718 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_07457cc5a3664344ae16b8a6fcb491af" } new_table_name: "default.Table_09ad2122f20d43989c7161f291a49a4b" modify_external_catalogs: true
I20251023 20:15:16.353057 23787 tablet_bootstrap.cc:492] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.354043 23787 ts_tablet_manager.cc:1403] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.003s
I20251023 20:15:16.354229 23785 ts_tablet_manager.cc:1929] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.354533 23785 log.cc:1199] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/dd456675077a4c73933c26e9a0313848
I20251023 20:15:16.355079 23785 ts_tablet_manager.cc:1950] T dd456675077a4c73933c26e9a0313848 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.355777 23787 raft_consensus.cc:359] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:16.357524 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d1ddde7e3134e23b3a9356523a93753 has no consensus state
I20251023 20:15:16.357606 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_eb3f472232804ea98a49fe822c8e9118" } new_table_name: "default.Table_8310cc42e75240cf843814583db61948" modify_external_catalogs: true
I20251023 20:15:16.358253 23366 tablet_service.cc:1505] Processing CreateTablet for tablet dafb9db710f94c328186e3179bf0b319 (DEFAULT_TABLE table=default.Table_82f6a78855d1488b9a8085c3bfbebab3 [id=f033b532f86d452c8f14e7f40ead8d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.358657 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafb9db710f94c328186e3179bf0b319. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.359774 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0898b76943db42e98d75a6e39cb8fade" } new_table_name: "default.Table_363cfc5e59804bc68a1148ba32dd13b1" modify_external_catalogs: true
I20251023 20:15:16.356000 23787 raft_consensus.cc:385] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.365216 23787 raft_consensus.cc:740] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.368818 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b7a18af7518d43c6b31ac4abed6372c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.369400 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7a18af7518d43c6b31ac4abed6372c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.369323 23787 consensus_queue.cc:260] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:16.370981 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 040a44e272b7410c98c109fbceb19192 consensus state has no leader
I20251023 20:15:16.370918 23508 tablet_service.cc:1505] Processing CreateTablet for tablet dafb9db710f94c328186e3179bf0b319 (DEFAULT_TABLE table=default.Table_82f6a78855d1488b9a8085c3bfbebab3 [id=f033b532f86d452c8f14e7f40ead8d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.371353 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dafb9db710f94c328186e3179bf0b319. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.372630 23787 ts_tablet_manager.cc:1434] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20251023 20:15:16.375769 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet c3035573f53c4a6db0bb72882ca84f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.376159 23783 tablet_replica.cc:333] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.376322 23783 raft_consensus.cc:2243] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.376509 23783 raft_consensus.cc:2272] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.376935 23637 tablet_service.cc:1505] Processing CreateTablet for tablet e5368813164949ff80e27bf59dcc931a (DEFAULT_TABLE table=default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.377357 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5368813164949ff80e27bf59dcc931a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.377470 23783 ts_tablet_manager.cc:1916] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.378329 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_20ccdc7f398e41459cbbb75ffa904e18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.378963 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20ccdc7f398e41459cbbb75ffa904e18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.380079 23790 tablet_bootstrap.cc:492] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.380230 23790 ts_tablet_manager.cc:1403] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20251023 20:15:16.380168 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_50879e22af484b719900c9b120534de0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.380641 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50879e22af484b719900c9b120534de0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.381064 23790 raft_consensus.cc:359] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.381289 23790 raft_consensus.cc:385] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.381384 23790 raft_consensus.cc:740] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.381618 23790 consensus_queue.cc:260] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.381824 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0f114a37952f447ab986c5799d7ff3ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.382146 23790 ts_tablet_manager.cc:1434] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20251023 20:15:16.382254 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f114a37952f447ab986c5799d7ff3ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.382622 23790 tablet_bootstrap.cc:492] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.383795 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_3220c87e07554676ba34368830409e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.384213 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3220c87e07554676ba34368830409e89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.384286 23790 tablet_bootstrap.cc:654] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.385658 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet 799132aedc784de3bf0fc6478c8ef496 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.388350 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 536216374bae4bc481d78fabbed3567f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.392041 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 8da82e6d24b14a129b91bb5298139d07 (DEFAULT_TABLE table=default.Table_58731fce9c224ff0b59d4b241110e518 [id=9ec90b724d89464a90821dd52282f77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.392438 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8da82e6d24b14a129b91bb5298139d07. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.392601 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e72922aae98430da968874f44603317 consensus state has no leader
I20251023 20:15:16.393824 23371 tablet_service.cc:1505] Processing CreateTablet for tablet e5368813164949ff80e27bf59dcc931a (DEFAULT_TABLE table=default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.393934 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ce2d16380766441fa6fbd45c614d640a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.394227 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5368813164949ff80e27bf59dcc931a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.394420 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce2d16380766441fa6fbd45c614d640a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:16.397740 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0f8d8184a68453abd2792ccc7740aa1 consensus state has no leader
I20251023 20:15:16.398001 23501 tablet_service.cc:1505] Processing CreateTablet for tablet e5368813164949ff80e27bf59dcc931a (DEFAULT_TABLE table=default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.398483 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5368813164949ff80e27bf59dcc931a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.399066 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (DEFAULT_TABLE table=default.Table_11159791d68d4dfb98fe2e8d7bb3afdc [id=b0c6ab2358b84be2862d31f863e3c32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.399438 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e76f2e26dcd4cdfbf72de899f5a1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.399619 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 8da82e6d24b14a129b91bb5298139d07 (DEFAULT_TABLE table=default.Table_58731fce9c224ff0b59d4b241110e518 [id=9ec90b724d89464a90821dd52282f77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.399971 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8da82e6d24b14a129b91bb5298139d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.404327 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet e31aebfc87734ffeb2c0b25c07cb3f70 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.405877 23783 ts_tablet_manager.cc:1929] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.405990 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 8da82e6d24b14a129b91bb5298139d07 (DEFAULT_TABLE table=default.Table_58731fce9c224ff0b59d4b241110e518 [id=9ec90b724d89464a90821dd52282f77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.406806 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8da82e6d24b14a129b91bb5298139d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.410275 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 2d698ffe7d154ef8b1994ffdd4b0c648 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.410807 23783 log.cc:1199] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/c3035573f53c4a6db0bb72882ca84f37
I20251023 20:15:16.411372 23783 ts_tablet_manager.cc:1950] T c3035573f53c4a6db0bb72882ca84f37 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:16.411775 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (DEFAULT_TABLE table=default.Table_11159791d68d4dfb98fe2e8d7bb3afdc [id=b0c6ab2358b84be2862d31f863e3c32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.412251 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (DEFAULT_TABLE table=default.Table_11159791d68d4dfb98fe2e8d7bb3afdc [id=b0c6ab2358b84be2862d31f863e3c32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.412758 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e76f2e26dcd4cdfbf72de899f5a1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.413389 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e76f2e26dcd4cdfbf72de899f5a1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.406486 23787 tablet_bootstrap.cc:492] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.414433 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet c3035573f53c4a6db0bb72882ca84f37 (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:16.416002 23787 tablet_bootstrap.cc:654] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.417498 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b6185784138b4c4b938effd9e433c447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.418093 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6185784138b4c4b938effd9e433c447 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.422842 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet dd456675077a4c73933c26e9a0313848 (table default.Table_da0691e8e6224cfebf17fe7df75954dd [id=a9e1acb22d6f423f924ad7417f9a986e]) successfully deleted
W20251023 20:15:16.423506 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 955412de8bd1482ea7304872b48f38c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 955412de8bd1482ea7304872b48f38c0 already in progress: creating tablet
I20251023 20:15:16.425380 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d7117a4238804d35b9c887c9d50900f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.425902 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7117a4238804d35b9c887c9d50900f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.426621 23785 tablet_replica.cc:333] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.426796 23785 raft_consensus.cc:2243] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.426965 23785 raft_consensus.cc:2272] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.429652 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet e31aebfc87734ffeb2c0b25c07cb3f70
I20251023 20:15:16.430233 23146 catalog_manager.cc:6719] ReplaceTablet: tablet e31aebfc87734ffeb2c0b25c07cb3f70 deleted and replaced by tablet edec01a63a3046d9ae5a372fc9068b52
I20251023 20:15:16.431288 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 with delete_type TABLET_DATA_DELETED (replaced by tablet edec01a63a3046d9ae5a372fc9068b52) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.432114 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 with delete_type TABLET_DATA_DELETED (replaced by tablet edec01a63a3046d9ae5a372fc9068b52) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.432358 23785 ts_tablet_manager.cc:1916] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:16.431195 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d698ffe7d154ef8b1994ffdd4b0c648 already in progress: creating tablet
W20251023 20:15:16.433530 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 23225d31c2e647398aa58494c627507e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23225d31c2e647398aa58494c627507e already in progress: creating tablet
W20251023 20:15:16.433751 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet dafb9db710f94c328186e3179bf0b319 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dafb9db710f94c328186e3179bf0b319 already in progress: creating tablet
W20251023 20:15:16.434435 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 5416da30c66645c19d36112980608527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5416da30c66645c19d36112980608527 already in progress: creating tablet
W20251023 20:15:16.434682 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9fc7e18054c440098dc2470143ea7fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fc7e18054c440098dc2470143ea7fae already in progress: creating tablet
I20251023 20:15:16.435839 23787 tablet_bootstrap.cc:492] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.436000 23787 ts_tablet_manager.cc:1403] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20251023 20:15:16.436817 23787 raft_consensus.cc:359] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.437000 23787 raft_consensus.cc:385] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.437186 23787 raft_consensus.cc:740] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.437563 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 with delete_type TABLET_DATA_DELETED (replaced by tablet edec01a63a3046d9ae5a372fc9068b52) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.437466 23787 consensus_queue.cc:260] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.438301 23787 ts_tablet_manager.cc:1434] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:16.437726 23786 tablet_bootstrap.cc:492] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.440636 23786 ts_tablet_manager.cc:1403] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.139s user 0.012s sys 0.005s
I20251023 20:15:16.441004 23784 ts_tablet_manager.cc:1929] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.441387 23787 tablet_bootstrap.cc:492] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.443253 23787 tablet_bootstrap.cc:654] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.443997 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d73da547fd04557abd5aba73e76dd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.444680 23786 raft_consensus.cc:359] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.445972 23786 raft_consensus.cc:385] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.448346 23786 raft_consensus.cc:740] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.448688 23786 consensus_queue.cc:260] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.449946 23786 ts_tablet_manager.cc:1434] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251023 20:15:16.451248 23786 tablet_bootstrap.cc:492] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.451224 23785 ts_tablet_manager.cc:1929] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.451346 23785 log.cc:1199] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/b09c0412f1eb48df8de542ca687a7b48
I20251023 20:15:16.453059 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 799132aedc784de3bf0fc6478c8ef496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.453236 23786 tablet_bootstrap.cc:654] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.453253 23784 log.cc:1199] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d8ad21e3888a4915ae5f252c2fca64cb
I20251023 20:15:16.453750 23784 ts_tablet_manager.cc:1950] T d8ad21e3888a4915ae5f252c2fca64cb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.454155 23785 ts_tablet_manager.cc:1950] T b09c0412f1eb48df8de542ca687a7b48 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:16.454687 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e31aebfc87734ffeb2c0b25c07cb3f70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e31aebfc87734ffeb2c0b25c07cb3f70 already in progress: creating tablet
I20251023 20:15:16.454950 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d8ad21e3888a4915ae5f252c2fca64cb (table default.Table_57a2a81ca5484113b5e2a0857d40dbbb [id=fc58c27a539f4454acbba8fbf3b62461]) successfully deleted
I20251023 20:15:16.455083 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet b09c0412f1eb48df8de542ca687a7b48 (table default.Table_45640565c4a24ff88274bc4e62b865ff [id=18ee017a8d484c7b9ff7dbbceaa5f7ac]) successfully deleted
I20251023 20:15:16.455415 23783 tablet_replica.cc:333] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.456315 23785 tablet_replica.cc:333] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.456502 23785 raft_consensus.cc:2243] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.456560 23783 raft_consensus.cc:2243] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.456681 23785 raft_consensus.cc:2272] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.456863 23783 raft_consensus.cc:2272] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.457731 23783 ts_tablet_manager.cc:1916] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.458168 23785 ts_tablet_manager.cc:1916] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.462666 23637 tablet_service.cc:1505] Processing CreateTablet for tablet f1a6fe726ef844829a7ffec9d4494637 (DEFAULT_TABLE table=default.Table_20ccdc7f398e41459cbbb75ffa904e18 [id=8d5fb738eb684e27860e03c12802a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.464311 23508 tablet_service.cc:1505] Processing CreateTablet for tablet f1a6fe726ef844829a7ffec9d4494637 (DEFAULT_TABLE table=default.Table_20ccdc7f398e41459cbbb75ffa904e18 [id=8d5fb738eb684e27860e03c12802a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.464762 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a6fe726ef844829a7ffec9d4494637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.464990 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c3d2d37dd2bd414fae4c968fc3f52a25" } modify_external_catalogs: true
I20251023 20:15:16.465237 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c3d2d37dd2bd414fae4c968fc3f52a25" } modify_external_catalogs: true
I20251023 20:15:16.465301 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4fba73266b6d4d36958e8f0566eb624e" } modify_external_catalogs: true
I20251023 20:15:16.465432 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4fba73266b6d4d36958e8f0566eb624e" } modify_external_catalogs: true
I20251023 20:15:16.468659 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 215e21ca449c481d8716783a65105101 (DEFAULT_TABLE table=default.Table_b7a18af7518d43c6b31ac4abed6372c2 [id=b68a3df9df3b49379181ef2b7573e973]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.469384 23371 tablet_service.cc:1505] Processing CreateTablet for tablet f1a6fe726ef844829a7ffec9d4494637 (DEFAULT_TABLE table=default.Table_20ccdc7f398e41459cbbb75ffa904e18 [id=8d5fb738eb684e27860e03c12802a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.469976 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a6fe726ef844829a7ffec9d4494637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.469918 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 215e21ca449c481d8716783a65105101 (DEFAULT_TABLE table=default.Table_b7a18af7518d43c6b31ac4abed6372c2 [id=b68a3df9df3b49379181ef2b7573e973]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.470536 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215e21ca449c481d8716783a65105101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.470927 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 (DEFAULT_TABLE table=default.Table_d8c675f452ec4b1ea9609eec6926317b [id=be3ef152e4fd4281b5caf942c8d404ab]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:16.471123 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 949e31d4a9ee4579843ed54153e22c7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 949e31d4a9ee4579843ed54153e22c7d already in progress: creating tablet
I20251023 20:15:16.471432 23784 tablet_replica.cc:333] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.471626 23784 raft_consensus.cc:2243] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.471787 23784 raft_consensus.cc:2272] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.471980 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 (DEFAULT_TABLE table=default.Table_d8c675f452ec4b1ea9609eec6926317b [id=be3ef152e4fd4281b5caf942c8d404ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.472580 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet e4bb6abe3ef848e2831839de6a28eb0f from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.473294 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fbad98d3ab24cc5b7726a92fba04cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.473412 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 215e21ca449c481d8716783a65105101 (DEFAULT_TABLE table=default.Table_b7a18af7518d43c6b31ac4abed6372c2 [id=b68a3df9df3b49379181ef2b7573e973]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.473758 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215e21ca449c481d8716783a65105101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.474350 23648 tablet_service.cc:1505] Processing CreateTablet for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (DEFAULT_TABLE table=default.Table_220abf897e5a4a07bd1af3ab929cb2d8 [id=276b2d82a9454bdc83331ebbcceed582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.474735 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 (DEFAULT_TABLE table=default.Table_d8c675f452ec4b1ea9609eec6926317b [id=be3ef152e4fd4281b5caf942c8d404ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.475135 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fbad98d3ab24cc5b7726a92fba04cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.475613 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet dc8f615c621043da88d52f3964b0a0ec
I20251023 20:15:16.475770 23505 tablet_service.cc:1505] Processing CreateTablet for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (DEFAULT_TABLE table=default.Table_220abf897e5a4a07bd1af3ab929cb2d8 [id=276b2d82a9454bdc83331ebbcceed582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.476097 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcbacd8e23bc4b129757d9faf89ce5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.476701 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet dc8f615c621043da88d52f3964b0a0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.479732 23790 tablet_bootstrap.cc:492] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.480661 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 585a9af140d9479fb83d13e68a62300c
I20251023 20:15:16.481261 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet dc8f615c621043da88d52f3964b0a0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.481947 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.482170 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.482905 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a6fe726ef844829a7ffec9d4494637. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.483494 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcbacd8e23bc4b129757d9faf89ce5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.483656 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.483956 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fbad98d3ab24cc5b7726a92fba04cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.484385 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215e21ca449c481d8716783a65105101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.485730 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet e4bb6abe3ef848e2831839de6a28eb0f
I20251023 20:15:16.486037 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet dc8f615c621043da88d52f3964b0a0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.486160 23141 catalog_manager.cc:6719] ReplaceTablet: tablet e4bb6abe3ef848e2831839de6a28eb0f deleted and replaced by tablet 8289ce37fa074891beb584bf2fff9747
I20251023 20:15:16.486243 23377 tablet_service.cc:1505] Processing CreateTablet for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (DEFAULT_TABLE table=default.Table_220abf897e5a4a07bd1af3ab929cb2d8 [id=276b2d82a9454bdc83331ebbcceed582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.486590 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcbacd8e23bc4b129757d9faf89ce5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.487161 23790 ts_tablet_manager.cc:1403] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.002s
I20251023 20:15:16.487924 23790 raft_consensus.cc:359] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.488111 23790 raft_consensus.cc:385] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.488186 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f with delete_type TABLET_DATA_DELETED (replaced by tablet 8289ce37fa074891beb584bf2fff9747) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.488562 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6f178deaeeb94c4da6a40568c635e0b5" } modify_external_catalogs: true
I20251023 20:15:16.488711 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6f178deaeeb94c4da6a40568c635e0b5" } modify_external_catalogs: true
I20251023 20:15:16.489923 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3320634019ac4bf48649eff33160d808" } modify_external_catalogs: true
I20251023 20:15:16.490103 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3320634019ac4bf48649eff33160d808" } modify_external_catalogs: true
I20251023 20:15:16.490132 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f with delete_type TABLET_DATA_DELETED (replaced by tablet 8289ce37fa074891beb584bf2fff9747) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.490389 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f with delete_type TABLET_DATA_DELETED (replaced by tablet 8289ce37fa074891beb584bf2fff9747) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.488188 23790 raft_consensus.cc:740] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.492245 23790 consensus_queue.cc:260] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.496675 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 1bcdec7c16c943079e20bb486aecb431
W20251023 20:15:16.497730 23134 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet e0f8d8184a68453abd2792ccc7740aa1 (table default.Table_363cfc5e59804bc68a1148ba32dd13b1 [id=936e505ef2f7447db8f9cb85af99c6d8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e0f8d8184a68453abd2792ccc7740aa1 consensus state has no leader
I20251023 20:15:16.498342 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bcdec7c16c943079e20bb486aecb431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.502499 23790 ts_tablet_manager.cc:1434] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251023 20:15:16.502978 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bcdec7c16c943079e20bb486aecb431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.503285 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bcdec7c16c943079e20bb486aecb431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.506335 23790 tablet_bootstrap.cc:492] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.507128 23784 ts_tablet_manager.cc:1916] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.508060 23790 tablet_bootstrap.cc:654] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.512059 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ce2d16380766441fa6fbd45c614d640a" } modify_external_catalogs: true
I20251023 20:15:16.512279 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ce2d16380766441fa6fbd45c614d640a" } modify_external_catalogs: true
I20251023 20:15:16.513238 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 4ea63540149c4c02abb8ae51209060a3 (DEFAULT_TABLE table=default.Table_e8cec02c09a44acab531a2f71ebb0a73 [id=41b1a0d78a924f2782bcfd5695f835d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.513615 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea63540149c4c02abb8ae51209060a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.515718 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69
I20251023 20:15:16.515791 23784 ts_tablet_manager.cc:1929] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.516963 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.517913 23787 tablet_bootstrap.cc:492] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.518122 23787 ts_tablet_manager.cc:1403] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.002s
I20251023 20:15:16.518909 23787 raft_consensus.cc:359] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.519183 23787 raft_consensus.cc:385] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.519294 23787 raft_consensus.cc:740] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.519280 23785 ts_tablet_manager.cc:1929] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.519434 23785 log.cc:1199] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/536216374bae4bc481d78fabbed3567f
I20251023 20:15:16.519870 23785 ts_tablet_manager.cc:1950] T 536216374bae4bc481d78fabbed3567f P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.519776 23787 consensus_queue.cc:260] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.520372 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.517242 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.520928 23784 log.cc:1199] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/2dfa9b2e447a4f6f9eca48f13a2dcd17
I20251023 20:15:16.521296 23784 ts_tablet_manager.cc:1950] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.522609 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 (table default.Table_579a7389d3174efdbaf6d1724548bf04 [id=f96dd558958e41a5a9dbf10177fe1277]) successfully deleted
I20251023 20:15:16.522982 23784 tablet_replica.cc:333] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.523499 23787 ts_tablet_manager.cc:1434] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:16.523797 23784 raft_consensus.cc:2243] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.524021 23784 raft_consensus.cc:2272] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.524673 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet bd41c9c84d814f9191d5952e71e31a06 with delete_type TABLET_DATA_DELETED (replaced by tablet 34a738510f514b088469de3d8a39ca2c) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.526259 23784 ts_tablet_manager.cc:1916] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.526746 23787 tablet_bootstrap.cc:492] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.527050 23140 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 90d7ab1529e54f6282e9abc60710df36
I20251023 20:15:16.528316 23787 tablet_bootstrap.cc:654] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.529215 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet b125422cf17145d5aef418817082eb32 with delete_type TABLET_DATA_DELETED (replaced by tablet e44dfb4fd3d04a58b5fcc6102e055ff1) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.530259 23785 tablet_replica.cc:333] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.530298 23786 tablet_bootstrap.cc:492] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.530432 23785 raft_consensus.cc:2243] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.530561 23786 ts_tablet_manager.cc:1403] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.080s user 0.000s sys 0.003s
I20251023 20:15:16.530623 23785 raft_consensus.cc:2272] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.530577 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_654a302882be43cd867d1b1ad3dc8bce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.531282 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_654a302882be43cd867d1b1ad3dc8bce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.531927 23786 raft_consensus.cc:359] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.532116 23786 raft_consensus.cc:385] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.532210 23786 raft_consensus.cc:740] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.532469 23786 consensus_queue.cc:260] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.530534 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7b49c2bed47942a1b7cb25f8675a69ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.532986 23786 ts_tablet_manager.cc:1434] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251023 20:15:16.533653 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b49c2bed47942a1b7cb25f8675a69ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.534821 23785 ts_tablet_manager.cc:1916] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.537956 23787 tablet_bootstrap.cc:492] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.538110 23787 ts_tablet_manager.cc:1403] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251023 20:15:16.538975 23787 raft_consensus.cc:359] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.539144 23787 raft_consensus.cc:385] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.539216 23787 raft_consensus.cc:740] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.539422 23787 consensus_queue.cc:260] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.542050 23784 ts_tablet_manager.cc:1929] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.542464 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 536216374bae4bc481d78fabbed3567f (table default.Table_50a809e112244d0984bf222990b09883 [id=5991f18468ca47ac82800d008795e0dc]) successfully deleted
I20251023 20:15:16.542706 23784 log.cc:1199] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/0d73da547fd04557abd5aba73e76dd18
W20251023 20:15:16.542759 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e31aebfc87734ffeb2c0b25c07cb3f70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e31aebfc87734ffeb2c0b25c07cb3f70 already in progress: creating tablet
W20251023 20:15:16.542961 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 585a9af140d9479fb83d13e68a62300c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585a9af140d9479fb83d13e68a62300c already in progress: creating tablet
W20251023 20:15:16.543161 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet dc8f615c621043da88d52f3964b0a0ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc8f615c621043da88d52f3964b0a0ec already in progress: creating tablet
W20251023 20:15:16.543337 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e4bb6abe3ef848e2831839de6a28eb0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4bb6abe3ef848e2831839de6a28eb0f already in progress: creating tablet
I20251023 20:15:16.543475 23784 ts_tablet_manager.cc:1950] T 0d73da547fd04557abd5aba73e76dd18 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.543510 23787 ts_tablet_manager.cc:1434] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20251023 20:15:16.544138 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1bcdec7c16c943079e20bb486aecb431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bcdec7c16c943079e20bb486aecb431 already in progress: creating tablet
W20251023 20:15:16.544389 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 already in progress: creating tablet
I20251023 20:15:16.544824 23785 ts_tablet_manager.cc:1929] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.544998 23785 log.cc:1199] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/bd41c9c84d814f9191d5952e71e31a06
I20251023 20:15:16.545471 23785 ts_tablet_manager.cc:1950] T bd41c9c84d814f9191d5952e71e31a06 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.546307 23787 tablet_bootstrap.cc:492] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.546958 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet bd41c9c84d814f9191d5952e71e31a06 (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]) successfully deleted
I20251023 20:15:16.547247 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.547271 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6cf378e7dc9b4a37acaec754f2edc359" } new_table_name: "default.Table_e714315a6ec44f9c81f1466e40cfd68f" modify_external_catalogs: true
I20251023 20:15:16.547948 23787 tablet_bootstrap.cc:654] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.552436 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 4ea63540149c4c02abb8ae51209060a3 (DEFAULT_TABLE table=default.Table_e8cec02c09a44acab531a2f71ebb0a73 [id=41b1a0d78a924f2782bcfd5695f835d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.552944 23501 tablet_service.cc:1505] Processing CreateTablet for tablet cff337a6e728489b99fa8c5d4dfece2b (DEFAULT_TABLE table=default.Table_94257ba5c56c4269842dcd5dd0d19461 [id=1abfa2f299624b41be063c4934d88e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.553032 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 4ea63540149c4c02abb8ae51209060a3 (DEFAULT_TABLE table=default.Table_e8cec02c09a44acab531a2f71ebb0a73 [id=41b1a0d78a924f2782bcfd5695f835d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.553717 23786 tablet_bootstrap.cc:492] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.553718 23366 tablet_service.cc:1505] Processing CreateTablet for tablet cff337a6e728489b99fa8c5d4dfece2b (DEFAULT_TABLE table=default.Table_94257ba5c56c4269842dcd5dd0d19461 [id=1abfa2f299624b41be063c4934d88e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.554093 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff337a6e728489b99fa8c5d4dfece2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.555454 23786 tablet_bootstrap.cc:654] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.559620 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff337a6e728489b99fa8c5d4dfece2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.559898 23783 ts_tablet_manager.cc:1929] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.560045 23783 log.cc:1199] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/0d73da547fd04557abd5aba73e76dd18
I20251023 20:15:16.560376 23783 ts_tablet_manager.cc:1950] T 0d73da547fd04557abd5aba73e76dd18 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:16.561910 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d58bceecba442898a3d742fa49a4a2 consensus state has no leader
I20251023 20:15:16.562317 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea63540149c4c02abb8ae51209060a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.564348 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea63540149c4c02abb8ae51209060a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.564913 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 0d73da547fd04557abd5aba73e76dd18 (table default.Table_d592564ff9cd44ab86010457642460ec [id=8d6a2964df664438b8680d81e4f040aa]) successfully deleted
I20251023 20:15:16.566687 23784 tablet_replica.cc:333] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.566956 23784 raft_consensus.cc:2243] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.567116 23784 raft_consensus.cc:2272] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.567840 23784 ts_tablet_manager.cc:1916] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.568226 23637 tablet_service.cc:1505] Processing CreateTablet for tablet cff337a6e728489b99fa8c5d4dfece2b (DEFAULT_TABLE table=default.Table_94257ba5c56c4269842dcd5dd0d19461 [id=1abfa2f299624b41be063c4934d88e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.568670 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff337a6e728489b99fa8c5d4dfece2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.573284 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 0d73da547fd04557abd5aba73e76dd18 (table default.Table_d592564ff9cd44ab86010457642460ec [id=8d6a2964df664438b8680d81e4f040aa]) successfully deleted
W20251023 20:15:16.573573 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 23225d31c2e647398aa58494c627507e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23225d31c2e647398aa58494c627507e already in progress: creating tablet
W20251023 20:15:16.573804 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d698ffe7d154ef8b1994ffdd4b0c648 already in progress: creating tablet
W20251023 20:15:16.574003 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5416da30c66645c19d36112980608527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5416da30c66645c19d36112980608527 already in progress: creating tablet
W20251023 20:15:16.574198 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 955412de8bd1482ea7304872b48f38c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 955412de8bd1482ea7304872b48f38c0 already in progress: creating tablet
W20251023 20:15:16.574373 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet dafb9db710f94c328186e3179bf0b319 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dafb9db710f94c328186e3179bf0b319 already in progress: creating tablet
W20251023 20:15:16.574572 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9fc7e18054c440098dc2470143ea7fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fc7e18054c440098dc2470143ea7fae already in progress: creating tablet
W20251023 20:15:16.574782 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e31aebfc87734ffeb2c0b25c07cb3f70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e31aebfc87734ffeb2c0b25c07cb3f70 already in progress: creating tablet
I20251023 20:15:16.575757 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 (DEFAULT_TABLE table=default.Table_2ca7b39d3f2341fb88826d33392f1d13 [id=2146c6f03d014ccdbc874c613400a8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.576151 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7301d6121c9d4c8dbc642e895aa4e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.576179 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 (DEFAULT_TABLE table=default.Table_2ca7b39d3f2341fb88826d33392f1d13 [id=2146c6f03d014ccdbc874c613400a8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.576567 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7301d6121c9d4c8dbc642e895aa4e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.577605 23790 tablet_bootstrap.cc:492] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.577755 23790 ts_tablet_manager.cc:1403] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.001s
I20251023 20:15:16.578738 23790 raft_consensus.cc:359] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.578919 23790 raft_consensus.cc:385] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.578996 23790 raft_consensus.cc:740] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.579255 23790 consensus_queue.cc:260] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.581670 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 52bc660b9da54c5d984381a267e1efa2 (DEFAULT_TABLE table=default.Table_0cd3fe0039de476c813272d09a9ba564 [id=161b57ca7cf148899ea058eab98b780d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.582078 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bc660b9da54c5d984381a267e1efa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.582238 23790 ts_tablet_manager.cc:1434] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:16.583313 23790 tablet_bootstrap.cc:492] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.584398 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 (DEFAULT_TABLE table=default.Table_2ca7b39d3f2341fb88826d33392f1d13 [id=2146c6f03d014ccdbc874c613400a8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.584829 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7301d6121c9d4c8dbc642e895aa4e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.585548 23790 tablet_bootstrap.cc:654] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.586231 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 52bc660b9da54c5d984381a267e1efa2 (DEFAULT_TABLE table=default.Table_0cd3fe0039de476c813272d09a9ba564 [id=161b57ca7cf148899ea058eab98b780d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.586707 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bc660b9da54c5d984381a267e1efa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.595541 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 52bc660b9da54c5d984381a267e1efa2 (DEFAULT_TABLE table=default.Table_0cd3fe0039de476c813272d09a9ba564 [id=161b57ca7cf148899ea058eab98b780d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.595968 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52bc660b9da54c5d984381a267e1efa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.600149 23790 tablet_bootstrap.cc:492] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.600416 23790 ts_tablet_manager.cc:1403] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20251023 20:15:16.601234 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 3fbad98d3ab24cc5b7726a92fba04cf8 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.601570 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7bbb68c0efad486c9267e6330c620301" } new_table_name: "default.Table_a1332462902c4ea5971afeb4c3688865" modify_external_catalogs: true
I20251023 20:15:16.601671 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet 62acc395fb294965a1ab476d37f9fab1 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.602566 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d8c675f452ec4b1ea9609eec6926317b" } new_table_name: "default.Table_fad48a232deb4365956d805e803043b0" modify_external_catalogs: true
I20251023 20:15:16.602946 23784 ts_tablet_manager.cc:1929] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.602926 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_363dbff3b2a046b190cdf7ce1a8572d6" } new_table_name: "default.Table_2e38cdb29ca54bdca4c030118988b786" modify_external_catalogs: true
I20251023 20:15:16.603092 23784 log.cc:1199] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/799132aedc784de3bf0fc6478c8ef496
I20251023 20:15:16.603585 23784 ts_tablet_manager.cc:1950] T 799132aedc784de3bf0fc6478c8ef496 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.604583 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 799132aedc784de3bf0fc6478c8ef496 (table default.Table_a90cbb8972924d6aad10abd0327c9874 [id=b2d8f2d64c7e4668a8e274bcdc384754]) successfully deleted
I20251023 20:15:16.604914 23784 tablet_replica.cc:333] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.605093 23784 raft_consensus.cc:2243] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.605273 23784 raft_consensus.cc:2272] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.605988 23784 ts_tablet_manager.cc:1916] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.607342 23787 tablet_bootstrap.cc:492] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.607466 23787 ts_tablet_manager.cc:1403] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.001s
I20251023 20:15:16.608175 23787 raft_consensus.cc:359] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.608359 23787 raft_consensus.cc:385] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.608429 23787 raft_consensus.cc:740] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.608690 23787 consensus_queue.cc:260] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.612885 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 62acc395fb294965a1ab476d37f9fab1
I20251023 20:15:16.613415 23139 catalog_manager.cc:6719] ReplaceTablet: tablet 62acc395fb294965a1ab476d37f9fab1 deleted and replaced by tablet a27f192792cf41288d69ca15ab538472
I20251023 20:15:16.613390 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 3fbad98d3ab24cc5b7726a92fba04cf8
I20251023 20:15:16.613680 23790 raft_consensus.cc:359] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:16.613891 23790 raft_consensus.cc:385] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.613907 23140 catalog_manager.cc:6719] ReplaceTablet: tablet 3fbad98d3ab24cc5b7726a92fba04cf8 deleted and replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:16.615398 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 62acc395fb294965a1ab476d37f9fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a27f192792cf41288d69ca15ab538472) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.615790 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 62acc395fb294965a1ab476d37f9fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a27f192792cf41288d69ca15ab538472) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.615790 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:16.616349 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d1ddde7e3134e23b3a9356523a93753 consensus state has no leader
I20251023 20:15:16.616918 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 (DEFAULT_TABLE table=default.Table_7bbb68c0efad486c9267e6330c620301 [id=f27defb723394b4db220b278b92e6ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.617440 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1ddde7e3134e23b3a9356523a93753. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.617488 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cec51c54ea534b1da4b1d170e79d9f1c has no consensus state
I20251023 20:15:16.617838 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.618160 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.618528 23785 tablet_replica.cc:333] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.618726 23785 raft_consensus.cc:2243] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.618882 23785 raft_consensus.cc:2272] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.619686 23785 ts_tablet_manager.cc:1916] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:16.621938 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet dc8f615c621043da88d52f3964b0a0ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc8f615c621043da88d52f3964b0a0ec already in progress: creating tablet
I20251023 20:15:16.622308 23783 tablet_replica.cc:333] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.622484 23783 raft_consensus.cc:2243] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.622684 23783 raft_consensus.cc:2272] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:16.622658 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 585a9af140d9479fb83d13e68a62300c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585a9af140d9479fb83d13e68a62300c already in progress: creating tablet
W20251023 20:15:16.623248 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e4bb6abe3ef848e2831839de6a28eb0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4bb6abe3ef848e2831839de6a28eb0f already in progress: creating tablet
I20251023 20:15:16.623711 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:16.624045 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1bcdec7c16c943079e20bb486aecb431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bcdec7c16c943079e20bb486aecb431 already in progress: creating tablet
W20251023 20:15:16.624332 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 already in progress: creating tablet
I20251023 20:15:16.624763 23787 ts_tablet_manager.cc:1434] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20251023 20:15:16.625105 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 (DEFAULT_TABLE table=default.Table_a1332462902c4ea5971afeb4c3688865 [id=f27defb723394b4db220b278b92e6ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.625327 23787 tablet_bootstrap.cc:492] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.625559 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1ddde7e3134e23b3a9356523a93753. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.627118 23785 ts_tablet_manager.cc:1929] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.627256 23785 log.cc:1199] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/2dfa9b2e447a4f6f9eca48f13a2dcd17
I20251023 20:15:16.627641 23785 ts_tablet_manager.cc:1950] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.613955 23790 raft_consensus.cc:740] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.628973 23790 consensus_queue.cc:260] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:16.628752 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet c95db7b5b9fd46acae8ffa7119915907 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.630926 23787 tablet_bootstrap.cc:654] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.635529 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 62acc395fb294965a1ab476d37f9fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a27f192792cf41288d69ca15ab538472) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.636534 23790 ts_tablet_manager.cc:1434] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20251023 20:15:16.636633 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 (DEFAULT_TABLE table=default.Table_a1332462902c4ea5971afeb4c3688865 [id=f27defb723394b4db220b278b92e6ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.636938 23783 ts_tablet_manager.cc:1916] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.637023 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d1ddde7e3134e23b3a9356523a93753. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.637482 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 (table default.Table_579a7389d3174efdbaf6d1724548bf04 [id=f96dd558958e41a5a9dbf10177fe1277]) successfully deleted
I20251023 20:15:16.638298 23641 tablet_service.cc:1505] Processing CreateTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 (DEFAULT_TABLE table=default.Table_e186a061f84042cd82645b37beae1da1 [id=dc553d161a13467c99df49720181dba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.638798 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.640600 23790 tablet_bootstrap.cc:492] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.643927 23790 tablet_bootstrap.cc:654] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:16.644404 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fbad98d3ab24cc5b7726a92fba04cf8 already in progress: creating tablet
W20251023 20:15:16.646162 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 62acc395fb294965a1ab476d37f9fab1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62acc395fb294965a1ab476d37f9fab1 already in progress: creating tablet
I20251023 20:15:16.651538 23786 tablet_bootstrap.cc:492] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.651739 23786 ts_tablet_manager.cc:1403] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.001s
I20251023 20:15:16.651782 23784 ts_tablet_manager.cc:1929] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.651985 23784 log.cc:1199] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/dc8f615c621043da88d52f3964b0a0ec
I20251023 20:15:16.652419 23784 ts_tablet_manager.cc:1950] T dc8f615c621043da88d52f3964b0a0ec P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.652706 23786 raft_consensus.cc:359] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.652920 23786 raft_consensus.cc:385] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.653003 23786 raft_consensus.cc:740] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.653468 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet dc8f615c621043da88d52f3964b0a0ec (table default.Table_4fba73266b6d4d36958e8f0566eb624e [id=d7983b16809245a2a73b2a6cd79f6e29]) successfully deleted
I20251023 20:15:16.653867 23786 consensus_queue.cc:260] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.654482 23786 ts_tablet_manager.cc:1434] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:16.655252 23786 tablet_bootstrap.cc:492] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.658134 23787 tablet_bootstrap.cc:492] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.658797 23787 ts_tablet_manager.cc:1403] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.002s
I20251023 20:15:16.659992 23787 raft_consensus.cc:359] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.660269 23787 raft_consensus.cc:385] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.660410 23787 raft_consensus.cc:740] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.660769 23787 consensus_queue.cc:260] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.661428 23786 tablet_bootstrap.cc:654] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.661968 23790 tablet_bootstrap.cc:492] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.662128 23790 ts_tablet_manager.cc:1403] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20251023 20:15:16.662685 23787 ts_tablet_manager.cc:1434] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:16.662966 23790 raft_consensus.cc:359] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.663875 23787 tablet_bootstrap.cc:492] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.664976 23371 tablet_service.cc:1505] Processing CreateTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 (DEFAULT_TABLE table=default.Table_e186a061f84042cd82645b37beae1da1 [id=dc553d161a13467c99df49720181dba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.664984 23790 raft_consensus.cc:385] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.665446 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.665882 23501 tablet_service.cc:1505] Processing CreateTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 (DEFAULT_TABLE table=default.Table_e186a061f84042cd82645b37beae1da1 [id=dc553d161a13467c99df49720181dba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.666421 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.666477 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 (DEFAULT_TABLE table=default.Table_6d625b139835401587825737c53000e3 [id=2ede045acf474fe9a77d38eaed85a15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.666880 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33b5efb30b5c491c8eafe3ac25688843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.670964 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 (DEFAULT_TABLE table=default.Table_6d625b139835401587825737c53000e3 [id=2ede045acf474fe9a77d38eaed85a15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.671376 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33b5efb30b5c491c8eafe3ac25688843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.673344 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 (DEFAULT_TABLE table=default.Table_6d625b139835401587825737c53000e3 [id=2ede045acf474fe9a77d38eaed85a15a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.665441 23790 raft_consensus.cc:740] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.674165 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33b5efb30b5c491c8eafe3ac25688843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.674363 23790 consensus_queue.cc:260] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.675182 23641 tablet_service.cc:1505] Processing CreateTablet for tablet f4778436aed64b53adb2bab54bcf263b (DEFAULT_TABLE table=default.Table_3fd19e6c8ca54391bd33c1c235d72938 [id=46e81c2c4c164452980264259fcff377]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.675710 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4778436aed64b53adb2bab54bcf263b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.678948 23790 ts_tablet_manager.cc:1434] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20251023 20:15:16.682693 23790 tablet_bootstrap.cc:492] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:16.684967 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 585a9af140d9479fb83d13e68a62300c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585a9af140d9479fb83d13e68a62300c already in progress: creating tablet
I20251023 20:15:16.685180 23368 tablet_service.cc:1505] Processing CreateTablet for tablet f4778436aed64b53adb2bab54bcf263b (DEFAULT_TABLE table=default.Table_3fd19e6c8ca54391bd33c1c235d72938 [id=46e81c2c4c164452980264259fcff377]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:16.685293 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e4bb6abe3ef848e2831839de6a28eb0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4bb6abe3ef848e2831839de6a28eb0f already in progress: creating tablet
W20251023 20:15:16.685501 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1bcdec7c16c943079e20bb486aecb431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bcdec7c16c943079e20bb486aecb431 already in progress: creating tablet
I20251023 20:15:16.685635 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4778436aed64b53adb2bab54bcf263b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.685731 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 already in progress: creating tablet
W20251023 20:15:16.685945 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 62acc395fb294965a1ab476d37f9fab1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62acc395fb294965a1ab476d37f9fab1 already in progress: creating tablet
I20251023 20:15:16.686129 23783 ts_tablet_manager.cc:1929] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:16.686184 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fbad98d3ab24cc5b7726a92fba04cf8 already in progress: creating tablet
I20251023 20:15:16.686261 23783 log.cc:1199] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/b125422cf17145d5aef418817082eb32
I20251023 20:15:16.686632 23783 ts_tablet_manager.cc:1950] T b125422cf17145d5aef418817082eb32 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:16.687832 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet b125422cf17145d5aef418817082eb32 (table default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:16.688804 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 (DEFAULT_TABLE table=default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.689033 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 (DEFAULT_TABLE table=default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.689342 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283166bbd8004ce3bca9f2ecbdbcbf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.689556 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283166bbd8004ce3bca9f2ecbdbcbf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.689709 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 (DEFAULT_TABLE table=default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.691330 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283166bbd8004ce3bca9f2ecbdbcbf48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.695147 23501 tablet_service.cc:1505] Processing CreateTablet for tablet f4778436aed64b53adb2bab54bcf263b (DEFAULT_TABLE table=default.Table_3fd19e6c8ca54391bd33c1c235d72938 [id=46e81c2c4c164452980264259fcff377]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.695169 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 9c0aeac2f20c4ca48c130a0c851f290f (DEFAULT_TABLE table=default.Table_8722fa0b5b6e4386af6870ce0b1f7963 [id=a7821323b5954830ad19f8f771bfe6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.695631 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c0aeac2f20c4ca48c130a0c851f290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.696892 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4778436aed64b53adb2bab54bcf263b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.698606 23790 tablet_bootstrap.cc:654] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.699177 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 9c0aeac2f20c4ca48c130a0c851f290f (DEFAULT_TABLE table=default.Table_8722fa0b5b6e4386af6870ce0b1f7963 [id=a7821323b5954830ad19f8f771bfe6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.699559 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c0aeac2f20c4ca48c130a0c851f290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.700785 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 9c0aeac2f20c4ca48c130a0c851f290f (DEFAULT_TABLE table=default.Table_8722fa0b5b6e4386af6870ce0b1f7963 [id=a7821323b5954830ad19f8f771bfe6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.701167 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c0aeac2f20c4ca48c130a0c851f290f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.706827 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_2d1938985e9345eba1d4fb750b552e78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.707479 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d1938985e9345eba1d4fb750b552e78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.708808 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e95151c6b95f40328bc00ecbfcc67d1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.709172 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_9a30e9c9272f47eb8fccd5e48ce47c30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.709429 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e95151c6b95f40328bc00ecbfcc67d1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:16.709748 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a30e9c9272f47eb8fccd5e48ce47c30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.713516 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4740fc3fe4ac4927b7f776c980467298"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.713840 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_187505c662504914a73c39146522f45e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.714135 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4740fc3fe4ac4927b7f776c980467298 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.714635 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_bf99025eb142454fa0513fafcbe636aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.715108 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf99025eb142454fa0513fafcbe636aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.716221 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_888857831b2f4735a4d944a2318eae73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.708814 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_220abf897e5a4a07bd1af3ab929cb2d8" } new_table_name: "default.Table_5b25a14abbd84fbe84dc7edc07272c09" modify_external_catalogs: true
W20251023 20:15:16.716624 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_187505c662504914a73c39146522f45e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:16.716698 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_888857831b2f4735a4d944a2318eae73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:16.719658 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0232c33db59641b3af27bfd15cd7fb81 consensus state has no leader
I20251023 20:15:16.723161 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7a4278caa64d4373b42c8b865ea6c2d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.727124 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a4278caa64d4373b42c8b865ea6c2d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.714644 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 1bcdec7c16c943079e20bb486aecb431 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.731112 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_537d89cd0e14498085731d83eae39d0a" } modify_external_catalogs: true
I20251023 20:15:16.731298 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_537d89cd0e14498085731d83eae39d0a" } modify_external_catalogs: true
I20251023 20:15:16.738876 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3220c87e07554676ba34368830409e89" } modify_external_catalogs: true
I20251023 20:15:16.739112 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3220c87e07554676ba34368830409e89" } modify_external_catalogs: true
W20251023 20:15:16.742465 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcbacd8e23bc4b129757d9faf89ce5b4 consensus state has no leader
I20251023 20:15:16.746376 23147 master_service.cc:954] ReplaceTablet: received request to replace tablet bd88285bf0a94fad858a25004609131c from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.746567 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d668e9c8e1394c4486b749d97aa78127" } modify_external_catalogs: true
I20251023 20:15:16.746745 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d668e9c8e1394c4486b749d97aa78127" } modify_external_catalogs: true
I20251023 20:15:16.747586 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a4375532322e46288e9b2a62d0a1cdea" } modify_external_catalogs: true
I20251023 20:15:16.747732 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a4375532322e46288e9b2a62d0a1cdea" } modify_external_catalogs: true
I20251023 20:15:16.748332 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.748417 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_67a76ff63d4d411582ab8e10ac4f6924" } modify_external_catalogs: true
I20251023 20:15:16.748615 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_67a76ff63d4d411582ab8e10ac4f6924" } modify_external_catalogs: true
W20251023 20:15:16.750000 23134 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 because the tablet was not found. No further retry: Not found: Tablet not found: 2dfa9b2e447a4f6f9eca48f13a2dcd17
I20251023 20:15:16.751633 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7e026bfaccc4482dba743310ed0adb4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.753602 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e026bfaccc4482dba743310ed0adb4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.755604 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet cc5ce157b6ca42da90b051a184983bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.757202 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 7645c36c983f438f8b1223256fd3ad2c (DEFAULT_TABLE table=default.Table_654a302882be43cd867d1b1ad3dc8bce [id=c830d9df19924cb09bb38ff2d445b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.757834 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7645c36c983f438f8b1223256fd3ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.758428 23501 tablet_service.cc:1505] Processing CreateTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe (DEFAULT_TABLE table=default.Table_d7117a4238804d35b9c887c9d50900f4 [id=7bab3a0ea7c2419e88d292294d15ba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.758818 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ae19cc80194363b8a9c96fbafb4ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.760468 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:16.760903 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet bd88285bf0a94fad858a25004609131c
I20251023 20:15:16.761359 23147 catalog_manager.cc:6719] ReplaceTablet: tablet bd88285bf0a94fad858a25004609131c deleted and replaced by tablet c317dc31cb0448d985512e6ba98735be
I20251023 20:15:16.761994 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.762050 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 451955aee3aa4bcb82252bcefbbcaefe
I20251023 20:15:16.762120 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet bd88285bf0a94fad858a25004609131c with delete_type TABLET_DATA_DELETED (replaced by tablet c317dc31cb0448d985512e6ba98735be) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.763285 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 3894dd4feeea4c16861a6c9520059474
I20251023 20:15:16.765267 23787 tablet_bootstrap.cc:654] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.766211 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.767907 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.769973 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 7645c36c983f438f8b1223256fd3ad2c (DEFAULT_TABLE table=default.Table_654a302882be43cd867d1b1ad3dc8bce [id=c830d9df19924cb09bb38ff2d445b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.770577 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7645c36c983f438f8b1223256fd3ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.777472 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 091e54a94f2d4c40a0006c7e35f74823 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.777721 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 451955aee3aa4bcb82252bcefbbcaefe) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.777974 23368 tablet_service.cc:1505] Processing CreateTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe (DEFAULT_TABLE table=default.Table_d7117a4238804d35b9c887c9d50900f4 [id=7bab3a0ea7c2419e88d292294d15ba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.778493 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ae19cc80194363b8a9c96fbafb4ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.779218 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.780099 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet d9ebd537f271426a95af432ec9070ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.780647 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 24528de9d0a84eddb091dec4f1d2b4e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 451955aee3aa4bcb82252bcefbbcaefe) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.781311 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.781610 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet bd88285bf0a94fad858a25004609131c with delete_type TABLET_DATA_DELETED (replaced by tablet c317dc31cb0448d985512e6ba98735be) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.782121 23785 tablet_replica.cc:333] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.782156 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 7645c36c983f438f8b1223256fd3ad2c (DEFAULT_TABLE table=default.Table_654a302882be43cd867d1b1ad3dc8bce [id=c830d9df19924cb09bb38ff2d445b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.782301 23785 raft_consensus.cc:2243] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.782228 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e13d2545a0a54b91a8bcb64db45581cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.782513 23785 raft_consensus.cc:2272] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.782547 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7645c36c983f438f8b1223256fd3ad2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.782792 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e13d2545a0a54b91a8bcb64db45581cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.777982 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.783777 23785 ts_tablet_manager.cc:1916] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.786576 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.786849 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet bd88285bf0a94fad858a25004609131c with delete_type TABLET_DATA_DELETED (replaced by tablet c317dc31cb0448d985512e6ba98735be) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.787353 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.789409 23784 tablet_replica.cc:333] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.789552 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.789697 23784 raft_consensus.cc:2243] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.789877 23784 raft_consensus.cc:2272] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.790620 23784 ts_tablet_manager.cc:1916] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:16.793548 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 451955aee3aa4bcb82252bcefbbcaefe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 451955aee3aa4bcb82252bcefbbcaefe already in progress: creating tablet
I20251023 20:15:16.794234 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8722fa0b5b6e4386af6870ce0b1f7963" } new_table_name: "default.Table_3f90e08382b448498312f1fade68ae16" modify_external_catalogs: true
I20251023 20:15:16.794637 23785 ts_tablet_manager.cc:1929] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.794729 23783 tablet_replica.cc:333] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:16.794773 23785 log.cc:1199] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d9ebd537f271426a95af432ec9070ae5
I20251023 20:15:16.794919 23783 raft_consensus.cc:2243] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.795117 23783 raft_consensus.cc:2272] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.795157 23785 ts_tablet_manager.cc:1950] T d9ebd537f271426a95af432ec9070ae5 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.799028 23644 tablet_service.cc:1505] Processing CreateTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe (DEFAULT_TABLE table=default.Table_d7117a4238804d35b9c887c9d50900f4 [id=7bab3a0ea7c2419e88d292294d15ba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.799508 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ae19cc80194363b8a9c96fbafb4ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.804251 23784 ts_tablet_manager.cc:1929] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.807345 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 (DEFAULT_TABLE table=default.Table_3220c87e07554676ba34368830409e89 [id=85bda6747ffe49f1b5a123f41013c426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.808085 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4beebee06e5341a29c65d0c3cf985ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.810386 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 (DEFAULT_TABLE table=default.Table_3220c87e07554676ba34368830409e89 [id=85bda6747ffe49f1b5a123f41013c426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.810827 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4beebee06e5341a29c65d0c3cf985ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.811228 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d9ebd537f271426a95af432ec9070ae5 (table default.Table_bbb88a1064eb4a27aa71756ff68bb6db [id=197fa361f58a4b24b5b394eb7f43b4e9]) successfully deleted
I20251023 20:15:16.811637 23785 tablet_replica.cc:333] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.806510 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 (DEFAULT_TABLE table=default.Table_3220c87e07554676ba34368830409e89 [id=85bda6747ffe49f1b5a123f41013c426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.813191 23785 raft_consensus.cc:2243] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.807098 23786 tablet_bootstrap.cc:492] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.813405 23786 ts_tablet_manager.cc:1403] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.158s user 0.005s sys 0.000s
I20251023 20:15:16.813578 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4beebee06e5341a29c65d0c3cf985ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.814257 23786 raft_consensus.cc:359] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.814435 23786 raft_consensus.cc:385] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.814517 23786 raft_consensus.cc:740] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.814755 23786 consensus_queue.cc:260] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.815230 23786 ts_tablet_manager.cc:1434] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:16.815539 23786 tablet_bootstrap.cc:492] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.815897 23785 raft_consensus.cc:2272] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.817431 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet edec01a63a3046d9ae5a372fc9068b52
I20251023 20:15:16.818938 23785 ts_tablet_manager.cc:1916] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.820031 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.820900 23786 tablet_bootstrap.cc:654] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.813200 23784 log.cc:1199] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/cc5ce157b6ca42da90b051a184983bfe
I20251023 20:15:16.822301 23784 ts_tablet_manager.cc:1950] T cc5ce157b6ca42da90b051a184983bfe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:16.823357 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet cc5ce157b6ca42da90b051a184983bfe (table default.Table_bd046e3e4fd9439c8aabbcee605207e4 [id=8b2cfa043c8147098454bfead6809936]) successfully deleted
I20251023 20:15:16.823719 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:16.824175 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 4beebee06e5341a29c65d0c3cf985ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4beebee06e5341a29c65d0c3cf985ef4 already in progress: creating tablet
I20251023 20:15:16.824461 23784 tablet_replica.cc:333] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:16.824621 23784 raft_consensus.cc:2243] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.825424 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e714315a6ec44f9c81f1466e40cfd68f" } new_table_name: "default.Table_07cf9b48edc541d7999a884181db7e0d" modify_external_catalogs: true
I20251023 20:15:16.825532 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2ca7b39d3f2341fb88826d33392f1d13" } new_table_name: "default.Table_465a2fe41b184ff49857af876a59642b" modify_external_catalogs: true
I20251023 20:15:16.826823 23784 raft_consensus.cc:2272] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.827576 23784 ts_tablet_manager.cc:1916] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.836575 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet ef1229b9e0384a82811f244d2bf4c8ce
I20251023 20:15:16.836897 23147 master_service.cc:954] ReplaceTablet: received request to replace tablet e4bb6abe3ef848e2831839de6a28eb0f from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.820490 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.837791 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.841226 23794 raft_consensus.cc:493] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:16.841406 23794 raft_consensus.cc:515] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.841967 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.842084 23794 leader_election.cc:290] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:16.842303 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:16.842756 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0232c33db59641b3af27bfd15cd7fb81" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:16.843107 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0232c33db59641b3af27bfd15cd7fb81" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:16.843663 23470 leader_election.cc:343] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:16.844069 23471 leader_election.cc:343] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:16.844194 23471 leader_election.cc:304] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:16.844616 23794 raft_consensus.cc:2749] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:16.845067 23147 master_service.cc:954] ReplaceTablet: received request to replace tablet 1bcdec7c16c943079e20bb486aecb431 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:16.845919 23505 tablet_service.cc:1505] Processing CreateTablet for tablet edec01a63a3046d9ae5a372fc9068b52 (DEFAULT_TABLE table=default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.846100 23371 tablet_service.cc:1505] Processing CreateTablet for tablet edec01a63a3046d9ae5a372fc9068b52 (DEFAULT_TABLE table=default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.846585 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edec01a63a3046d9ae5a372fc9068b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.849025 23648 tablet_service.cc:1505] Processing CreateTablet for tablet edec01a63a3046d9ae5a372fc9068b52 (DEFAULT_TABLE table=default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.849498 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edec01a63a3046d9ae5a372fc9068b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.855170 23790 tablet_bootstrap.cc:492] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.855392 23790 ts_tablet_manager.cc:1403] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.173s user 0.013s sys 0.017s
I20251023 20:15:16.856797 23790 raft_consensus.cc:359] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.857614 23501 tablet_service.cc:1505] Processing CreateTablet for tablet a27f192792cf41288d69ca15ab538472 (DEFAULT_TABLE table=default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.857583 23371 tablet_service.cc:1505] Processing CreateTablet for tablet a27f192792cf41288d69ca15ab538472 (DEFAULT_TABLE table=default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.858099 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27f192792cf41288d69ca15ab538472. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.858501 23787 tablet_bootstrap.cc:492] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.858672 23787 ts_tablet_manager.cc:1403] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.195s user 0.003s sys 0.007s
I20251023 20:15:16.859431 23790 raft_consensus.cc:385] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.859372 23787 raft_consensus.cc:359] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.859553 23787 raft_consensus.cc:385] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.859625 23787 raft_consensus.cc:740] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:16.859645 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c0aeac2f20c4ca48c130a0c851f290f consensus state has no leader
I20251023 20:15:16.859639 23790 raft_consensus.cc:740] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.859890 23787 consensus_queue.cc:260] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.859936 23790 consensus_queue.cc:260] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.861264 23787 ts_tablet_manager.cc:1434] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:16.861562 23787 tablet_bootstrap.cc:492] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.863768 23787 tablet_bootstrap.cc:654] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.865684 23790 ts_tablet_manager.cc:1434] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251023 20:15:16.869176 23790 tablet_bootstrap.cc:492] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.869717 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 8289ce37fa074891beb584bf2fff9747 (DEFAULT_TABLE table=default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.869922 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 (DEFAULT_TABLE table=default.Table_0f114a37952f447ab986c5799d7ff3ca [id=6945386a3c8140c7b509b38c86326f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.870076 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 7540354f8577482785e6341fa9022315 (DEFAULT_TABLE table=default.Table_b6185784138b4c4b938effd9e433c447 [id=75026c75215e419983355275cf38f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.871838 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7540354f8577482785e6341fa9022315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.872170 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a2f3adb5d347a0adc198cd1ef93304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.872617 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8289ce37fa074891beb584bf2fff9747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.872596 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 8289ce37fa074891beb584bf2fff9747 (DEFAULT_TABLE table=default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.872994 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8289ce37fa074891beb584bf2fff9747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.873055 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edec01a63a3046d9ae5a372fc9068b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.877377 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 7540354f8577482785e6341fa9022315 (DEFAULT_TABLE table=default.Table_b6185784138b4c4b938effd9e433c447 [id=75026c75215e419983355275cf38f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.877700 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7540354f8577482785e6341fa9022315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.869565 23785 ts_tablet_manager.cc:1929] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.879556 23785 log.cc:1199] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/24528de9d0a84eddb091dec4f1d2b4e2
I20251023 20:15:16.879949 23785 ts_tablet_manager.cc:1950] T 24528de9d0a84eddb091dec4f1d2b4e2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.880947 23790 tablet_bootstrap.cc:654] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.870172 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 8289ce37fa074891beb584bf2fff9747 (DEFAULT_TABLE table=default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.870679 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f205f4e6d547229f3026c16a6fd485 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.871369 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 7540354f8577482785e6341fa9022315 (DEFAULT_TABLE table=default.Table_b6185784138b4c4b938effd9e433c447 [id=75026c75215e419983355275cf38f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.872949 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 (DEFAULT_TABLE table=default.Table_0f114a37952f447ab986c5799d7ff3ca [id=6945386a3c8140c7b509b38c86326f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.882121 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7540354f8577482785e6341fa9022315. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.882259 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 24528de9d0a84eddb091dec4f1d2b4e2 (table default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
W20251023 20:15:16.882586 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 3894dd4feeea4c16861a6c9520059474 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3894dd4feeea4c16861a6c9520059474 already in progress: creating tablet
W20251023 20:15:16.882794 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet bd88285bf0a94fad858a25004609131c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd88285bf0a94fad858a25004609131c already in progress: creating tablet
W20251023 20:15:16.883018 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 4beebee06e5341a29c65d0c3cf985ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4beebee06e5341a29c65d0c3cf985ef4 already in progress: creating tablet
W20251023 20:15:16.883210 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet edec01a63a3046d9ae5a372fc9068b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edec01a63a3046d9ae5a372fc9068b52 already in progress: creating tablet
W20251023 20:15:16.883433 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet ef1229b9e0384a82811f244d2bf4c8ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef1229b9e0384a82811f244d2bf4c8ce already in progress: creating tablet
W20251023 20:15:16.884271 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7301d6121c9d4c8dbc642e895aa4e9f6 consensus state has no leader
I20251023 20:15:16.884301 23783 ts_tablet_manager.cc:1916] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:16.884668 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d58bceecba442898a3d742fa49a4a2 consensus state has no leader
I20251023 20:15:16.889249 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a2f3adb5d347a0adc198cd1ef93304. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.892345 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27f192792cf41288d69ca15ab538472. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.894292 23786 tablet_bootstrap.cc:492] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:16.894464 23786 ts_tablet_manager.cc:1403] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.007s
I20251023 20:15:16.894996 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8289ce37fa074891beb584bf2fff9747. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.895231 23786 raft_consensus.cc:359] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.895423 23786 raft_consensus.cc:385] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.895490 23786 raft_consensus.cc:740] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.895759 23786 consensus_queue.cc:260] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.897724 23786 ts_tablet_manager.cc:1434] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:16.898150 23786 tablet_bootstrap.cc:492] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:16.899263 23785 tablet_replica.cc:333] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:16.899700 23785 raft_consensus.cc:2243] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:16.899894 23785 raft_consensus.cc:2272] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:16.900694 23785 ts_tablet_manager.cc:1916] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:16.902190 23374 tablet_service.cc:1505] Processing CreateTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f (DEFAULT_TABLE table=default.Table_7b49c2bed47942a1b7cb25f8675a69ea [id=8cf42619d4b44a5990f7f63f1e1315ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.902604 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.903074 23790 tablet_bootstrap.cc:492] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:16.905436 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 147781e046fe4feb9815147c774fe210 (DEFAULT_TABLE table=default.Table_50879e22af484b719900c9b120534de0 [id=1245153d2d05472a9f20bc69214ca3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.906181 23366 tablet_service.cc:1505] Processing CreateTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c (DEFAULT_TABLE table=default.Table_fad48a232deb4365956d805e803043b0 [id=be3ef152e4fd4281b5caf942c8d404ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.907183 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cec51c54ea534b1da4b1d170e79d9f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.911899 23801 raft_consensus.cc:493] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:16.911945 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.912086 23801 raft_consensus.cc:515] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.913250 23648 tablet_service.cc:1505] Processing CreateTablet for tablet a27f192792cf41288d69ca15ab538472 (DEFAULT_TABLE table=default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.913328 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8ad21e3888a4915ae5f252c2fca64cb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:16.914017 23801 leader_election.cc:290] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:16.915098 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 (DEFAULT_TABLE table=default.Table_0f114a37952f447ab986c5799d7ff3ca [id=6945386a3c8140c7b509b38c86326f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.915397 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147781e046fe4feb9815147c774fe210. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.915973 23650 tablet_service.cc:1505] Processing CreateTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f (DEFAULT_TABLE table=default.Table_7b49c2bed47942a1b7cb25f8675a69ea [id=8cf42619d4b44a5990f7f63f1e1315ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.916254 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27f192792cf41288d69ca15ab538472. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.916489 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a2f3adb5d347a0adc198cd1ef93304. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.917696 23605 leader_election.cc:343] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:16.918071 23639 tablet_service.cc:1505] Processing CreateTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c (DEFAULT_TABLE table=default.Table_fad48a232deb4365956d805e803043b0 [id=be3ef152e4fd4281b5caf942c8d404ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.921231 23790 ts_tablet_manager.cc:1403] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20251023 20:15:16.922084 23790 raft_consensus.cc:359] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.922282 23790 raft_consensus.cc:385] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.922359 23790 raft_consensus.cc:740] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.923812 23515 tablet_service.cc:1505] Processing CreateTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f (DEFAULT_TABLE table=default.Table_7b49c2bed47942a1b7cb25f8675a69ea [id=8cf42619d4b44a5990f7f63f1e1315ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.923903 23508 tablet_service.cc:1505] Processing CreateTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c (DEFAULT_TABLE table=default.Table_fad48a232deb4365956d805e803043b0 [id=be3ef152e4fd4281b5caf942c8d404ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.924577 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.925144 23604 leader_election.cc:343] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: d8ad21e3888a4915ae5f252c2fca64cb
I20251023 20:15:16.925431 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cec51c54ea534b1da4b1d170e79d9f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.925490 23790 consensus_queue.cc:260] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.926061 23604 leader_election.cc:304] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:16.926478 23801 raft_consensus.cc:2749] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:16.927568 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 147781e046fe4feb9815147c774fe210 (DEFAULT_TABLE table=default.Table_50879e22af484b719900c9b120534de0 [id=1245153d2d05472a9f20bc69214ca3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.927989 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147781e046fe4feb9815147c774fe210. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.932833 23790 ts_tablet_manager.cc:1434] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251023 20:15:16.934048 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:16.942808 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 523d72012c364457b9a2a299f2f789a1 (DEFAULT_TABLE table=default.Table_7e026bfaccc4482dba743310ed0adb4c [id=7fdd0d3b02734ea89c0fce5e98fca82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.943743 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 6e1b4748c0244b258960b1025da47562 (DEFAULT_TABLE table=default.Table_4740fc3fe4ac4927b7f776c980467298 [id=a93a658ff7104725a680f6ee3686e0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.944090 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1b4748c0244b258960b1025da47562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.946409 23374 tablet_service.cc:1505] Processing CreateTablet for tablet ae388da9e4f94ba5af82a67088ab7097 (DEFAULT_TABLE table=default.Table_7a4278caa64d4373b42c8b865ea6c2d8 [id=8f99b8c48cce4b63af995ef9ddb1edfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.946746 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae388da9e4f94ba5af82a67088ab7097. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.947751 23645 tablet_service.cc:1505] Processing CreateTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 (DEFAULT_TABLE table=default.Table_187505c662504914a73c39146522f45e [id=ebb0bfc95f844d50859f36a09ac81f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.948119 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21ec49194e24723b31fd9c86f1f85e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.949455 23646 tablet_service.cc:1505] Processing CreateTablet for tablet ae388da9e4f94ba5af82a67088ab7097 (DEFAULT_TABLE table=default.Table_7a4278caa64d4373b42c8b865ea6c2d8 [id=8f99b8c48cce4b63af995ef9ddb1edfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.949855 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae388da9e4f94ba5af82a67088ab7097. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.950076 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523d72012c364457b9a2a299f2f789a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.954319 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_44aa985f505a4135bf3a3f6db293c3eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.954417 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e926815d67914fed915b184838f05b72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.954581 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ea99b2e89c224754a8eba8255433cd14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.954654 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_1662565661d643d8b4ff35b3e84f0351" } new_table_name: "default.Table_3a893d487d5b46ca84e6d00ce6cf7137" modify_external_catalogs: true
W20251023 20:15:16.954905 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44aa985f505a4135bf3a3f6db293c3eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:16.954933 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea99b2e89c224754a8eba8255433cd14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.955212 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c709570632244bd3af291b4e143d2eb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.955535 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 6e1b4748c0244b258960b1025da47562 (DEFAULT_TABLE table=default.Table_4740fc3fe4ac4927b7f776c980467298 [id=a93a658ff7104725a680f6ee3686e0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.955732 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4e18a968658f470d870bc932102ebf00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.955801 23370 tablet_service.cc:1505] Processing CreateTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 (DEFAULT_TABLE table=default.Table_187505c662504914a73c39146522f45e [id=ebb0bfc95f844d50859f36a09ac81f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.955893 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1b4748c0244b258960b1025da47562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.956135 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21ec49194e24723b31fd9c86f1f85e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.956161 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e18a968658f470d870bc932102ebf00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.957239 23790 tablet_bootstrap.cc:492] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:16.954417 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c27155507a7f45cc860ea6089419e20e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:16.959957 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.960238 23379 tablet_service.cc:1505] Processing CreateTablet for tablet 244fa327fad74aad9e57a7fbcbf908bb (DEFAULT_TABLE table=default.Table_e95151c6b95f40328bc00ecbfcc67d1a [id=910ad9776933488eb34806b4aaff92d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.960592 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244fa327fad74aad9e57a7fbcbf908bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.962138 23790 tablet_bootstrap.cc:654] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.964214 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 0438593fbbdf47cebfb64121acad104a (DEFAULT_TABLE table=default.Table_e13d2545a0a54b91a8bcb64db45581cc [id=5d8522349b2648fda1e17a6fd1b301dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.964552 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0438593fbbdf47cebfb64121acad104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.964437 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_914e9b5e4a7443c5baf677b4b6857359"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.964941 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_914e9b5e4a7443c5baf677b4b6857359 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.965720 23785 ts_tablet_manager.cc:1929] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.965873 23785 log.cc:1199] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/73f205f4e6d547229f3026c16a6fd485
I20251023 20:15:16.966430 23785 ts_tablet_manager.cc:1950] T 73f205f4e6d547229f3026c16a6fd485 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:16.967253 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 0438593fbbdf47cebfb64121acad104a (DEFAULT_TABLE table=default.Table_e13d2545a0a54b91a8bcb64db45581cc [id=5d8522349b2648fda1e17a6fd1b301dd]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:16.968448 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c27155507a7f45cc860ea6089419e20e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.970216 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 523d72012c364457b9a2a299f2f789a1 (DEFAULT_TABLE table=default.Table_7e026bfaccc4482dba743310ed0adb4c [id=7fdd0d3b02734ea89c0fce5e98fca82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.970579 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523d72012c364457b9a2a299f2f789a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:16.970803 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e926815d67914fed915b184838f05b72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:16.971606 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c709570632244bd3af291b4e143d2eb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.973416 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 147781e046fe4feb9815147c774fe210 (DEFAULT_TABLE table=default.Table_50879e22af484b719900c9b120534de0 [id=1245153d2d05472a9f20bc69214ca3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.976634 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 6e1b4748c0244b258960b1025da47562 (DEFAULT_TABLE table=default.Table_4740fc3fe4ac4927b7f776c980467298 [id=a93a658ff7104725a680f6ee3686e0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.976418 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_363cfc5e59804bc68a1148ba32dd13b1" } new_table_name: "default.Table_d880d87e1b2244b8ba05b6ff6653e35b" modify_external_catalogs: true
I20251023 20:15:16.978331 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_cdf01488bd3e4c7c98f16522764f7a18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:16.978879 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdf01488bd3e4c7c98f16522764f7a18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:16.978911 23787 tablet_bootstrap.cc:492] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:16.979051 23787 ts_tablet_manager.cc:1403] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.002s
I20251023 20:15:16.979789 23787 raft_consensus.cc:359] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.979965 23787 raft_consensus.cc:385] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:16.980053 23787 raft_consensus.cc:740] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:16.980268 23787 consensus_queue.cc:260] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:16.980731 23787 ts_tablet_manager.cc:1434] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:16.981035 23787 tablet_bootstrap.cc:492] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:16.986394 23787 tablet_bootstrap.cc:654] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:16.988444 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 0438593fbbdf47cebfb64121acad104a (DEFAULT_TABLE table=default.Table_e13d2545a0a54b91a8bcb64db45581cc [id=5d8522349b2648fda1e17a6fd1b301dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.977455 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 523d72012c364457b9a2a299f2f789a1 (DEFAULT_TABLE table=default.Table_7e026bfaccc4482dba743310ed0adb4c [id=7fdd0d3b02734ea89c0fce5e98fca82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:16.989488 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0438593fbbdf47cebfb64121acad104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.989987 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cec51c54ea534b1da4b1d170e79d9f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:16.990479 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 216e2c887a364c8ea5068c3834d1e4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:16.994534 23783 ts_tablet_manager.cc:1929] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:16.996306 23783 log.cc:1199] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/2dfa9b2e447a4f6f9eca48f13a2dcd17
I20251023 20:15:16.996831 23783 ts_tablet_manager.cc:1950] T 2dfa9b2e447a4f6f9eca48f13a2dcd17 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:16.995962 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 244fa327fad74aad9e57a7fbcbf908bb (DEFAULT_TABLE table=default.Table_e95151c6b95f40328bc00ecbfcc67d1a [id=910ad9776933488eb34806b4aaff92d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:16.997047 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e418a62d00824cff89b37a7bace5c469 consensus state has no leader
I20251023 20:15:16.997457 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244fa327fad74aad9e57a7fbcbf908bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.006826 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 2dfa9b2e447a4f6f9eca48f13a2dcd17 (table default.Table_579a7389d3174efdbaf6d1724548bf04 [id=f96dd558958e41a5a9dbf10177fe1277]) successfully deleted
I20251023 20:15:17.008298 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 523d72012c364457b9a2a299f2f789a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.008580 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147781e046fe4feb9815147c774fe210. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.009042 23786 tablet_bootstrap.cc:654] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.009778 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 (DEFAULT_TABLE table=default.Table_2d1938985e9345eba1d4fb750b552e78 [id=8b537204d252421c8ef4084d1dd9226f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.009800 23503 tablet_service.cc:1505] Processing CreateTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 (DEFAULT_TABLE table=default.Table_187505c662504914a73c39146522f45e [id=ebb0bfc95f844d50859f36a09ac81f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.009953 23516 tablet_service.cc:1505] Processing CreateTablet for tablet ae388da9e4f94ba5af82a67088ab7097 (DEFAULT_TABLE table=default.Table_7a4278caa64d4373b42c8b865ea6c2d8 [id=8f99b8c48cce4b63af995ef9ddb1edfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.010309 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95574277dde44983bb0a43d2b5ad01d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:17.013021 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 62acc395fb294965a1ab476d37f9fab1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62acc395fb294965a1ab476d37f9fab1 already in progress: creating tablet
I20251023 20:15:17.014087 23787 tablet_bootstrap.cc:492] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:17.014087 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fbad98d3ab24cc5b7726a92fba04cf8 already in progress: creating tablet
I20251023 20:15:17.014382 23787 ts_tablet_manager.cc:1403] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
W20251023 20:15:17.015036 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0f8d8184a68453abd2792ccc7740aa1 consensus state has no leader
I20251023 20:15:17.015285 23787 raft_consensus.cc:359] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.015539 23787 raft_consensus.cc:385] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.015646 23787 raft_consensus.cc:740] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.011305 23785 tablet_replica.cc:333] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:17.017280 23785 raft_consensus.cc:2243] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.017752 23785 raft_consensus.cc:2272] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.018066 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 (DEFAULT_TABLE table=default.Table_2d1938985e9345eba1d4fb750b552e78 [id=8b537204d252421c8ef4084d1dd9226f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.018314 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 244fa327fad74aad9e57a7fbcbf908bb (DEFAULT_TABLE table=default.Table_e95151c6b95f40328bc00ecbfcc67d1a [id=910ad9776933488eb34806b4aaff92d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.018783 23785 ts_tablet_manager.cc:1916] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.019120 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0438593fbbdf47cebfb64121acad104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.017839 23787 consensus_queue.cc:260] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.020653 23783 tablet_replica.cc:333] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.021872 23787 ts_tablet_manager.cc:1434] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:17.021886 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2d1938985e9345eba1d4fb750b552e78" } modify_external_catalogs: true
I20251023 20:15:17.021798 23783 raft_consensus.cc:2243] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.022089 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2d1938985e9345eba1d4fb750b552e78" } modify_external_catalogs: true
I20251023 20:15:17.022360 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a1332462902c4ea5971afeb4c3688865" } modify_external_catalogs: true
I20251023 20:15:17.022524 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a1332462902c4ea5971afeb4c3688865" } modify_external_catalogs: true
I20251023 20:15:17.023175 23377 tablet_service.cc:1505] Processing CreateTablet for tablet c317dc31cb0448d985512e6ba98735be (DEFAULT_TABLE table=default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.023233 23783 raft_consensus.cc:2272] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.023699 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c317dc31cb0448d985512e6ba98735be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.024437 23787 tablet_bootstrap.cc:492] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.026044 23144 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998]). Expected version 4 got 0
I20251023 20:15:17.026260 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 (DEFAULT_TABLE table=default.Table_2d1938985e9345eba1d4fb750b552e78 [id=8b537204d252421c8ef4084d1dd9226f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.027655 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95574277dde44983bb0a43d2b5ad01d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.021886 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7b49c2bed47942a1b7cb25f8675a69ea" } modify_external_catalogs: true
I20251023 20:15:17.027912 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7b49c2bed47942a1b7cb25f8675a69ea" } modify_external_catalogs: true
I20251023 20:15:17.028940 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:17.029351 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bf99025eb142454fa0513fafcbe636aa" } modify_external_catalogs: true
I20251023 20:15:17.029481 23143 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998]). Expected version 4 got 0
I20251023 20:15:17.029580 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bf99025eb142454fa0513fafcbe636aa" } modify_external_catalogs: true
I20251023 20:15:17.031327 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.032214 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1b4748c0244b258960b1025da47562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.033669 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 73f205f4e6d547229f3026c16a6fd485 (table default.Table_43bf18200b234de68377af27f882f395 [id=da87b3d063a64aab92f939df02727144]) successfully deleted
I20251023 20:15:17.034935 23783 ts_tablet_manager.cc:1916] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.036175 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 9d1ddde7e3134e23b3a9356523a93753
I20251023 20:15:17.036978 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.037199 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:17.038571 23134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.127.190:34331: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251023 20:15:17.040503 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244fa327fad74aad9e57a7fbcbf908bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.041996 23639 tablet_service.cc:1505] Processing CreateTablet for tablet c317dc31cb0448d985512e6ba98735be (DEFAULT_TABLE table=default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.042469 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c317dc31cb0448d985512e6ba98735be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.045223 23785 ts_tablet_manager.cc:1929] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.045352 23785 log.cc:1199] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/216e2c887a364c8ea5068c3834d1e4e2
I20251023 20:15:17.045562 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae388da9e4f94ba5af82a67088ab7097. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.045677 23785 ts_tablet_manager.cc:1950] T 216e2c887a364c8ea5068c3834d1e4e2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:17.045562 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.046085 23790 tablet_bootstrap.cc:492] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.046254 23790 ts_tablet_manager.cc:1403] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.005s
I20251023 20:15:17.047030 23790 raft_consensus.cc:359] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.047228 23790 raft_consensus.cc:385] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.047299 23790 raft_consensus.cc:740] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.047555 23790 consensus_queue.cc:260] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.047760 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.048007 23790 ts_tablet_manager.cc:1434] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:17.048557 23506 tablet_service.cc:1505] Processing CreateTablet for tablet c317dc31cb0448d985512e6ba98735be (DEFAULT_TABLE table=default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.048213 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 (DEFAULT_TABLE table=default.Table_9a30e9c9272f47eb8fccd5e48ce47c30 [id=8b9eb030f1a24f68bb8e2e87b580727a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.051074 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e3d422a66f432789c0cf4a00e57af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.054775 23790 tablet_bootstrap.cc:492] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.050302 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 4f47ddde9f3d4042bc84853332e21950 (DEFAULT_TABLE table=default.Table_888857831b2f4735a4d944a2318eae73 [id=651489d40b6b425897b71f97e89c6dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.050366 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 2532c8567db14901b383f42e5f04b4c2 (DEFAULT_TABLE table=default.Table_bf99025eb142454fa0513fafcbe636aa [id=f52f1d83bd9345ffa08e6d3d282914e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.055591 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2532c8567db14901b383f42e5f04b4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.056751 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f47ddde9f3d4042bc84853332e21950. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.057287 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 2532c8567db14901b383f42e5f04b4c2 (DEFAULT_TABLE table=default.Table_bf99025eb142454fa0513fafcbe636aa [id=f52f1d83bd9345ffa08e6d3d282914e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.057660 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2532c8567db14901b383f42e5f04b4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.058301 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 216e2c887a364c8ea5068c3834d1e4e2 (table default.Table_d94baddda6104b5c9c007c95623ea0dd [id=ce02fa77823e4fbcad36a7e21bf10630]) successfully deleted
W20251023 20:15:17.058553 23134 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 95574277dde44983bb0a43d2b5ad01d5 because the tablet was not found. No further retry: Not found: Tablet not found: 95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:17.059002 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0cd3fe0039de476c813272d09a9ba564" } modify_external_catalogs: true
I20251023 20:15:17.059180 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0cd3fe0039de476c813272d09a9ba564" } modify_external_catalogs: true
I20251023 20:15:17.060601 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b3b5199d1faf4e5795ba391229812ea9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.060848 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 799132aedc784de3bf0fc6478c8ef496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:17.061208 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3b5199d1faf4e5795ba391229812ea9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.062019 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_da2bae6bea6047318cd9fb653b35475c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.062366 23787 tablet_bootstrap.cc:654] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:17.062491 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da2bae6bea6047318cd9fb653b35475c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.065032 23790 tablet_bootstrap.cc:654] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.067286 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 2532c8567db14901b383f42e5f04b4c2
I20251023 20:15:17.067914 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.068281 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.069123 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_44aa985f505a4135bf3a3f6db293c3eb" } new_table_name: "default.Table_1fd972f3964245fab674e2cfc26a8210" modify_external_catalogs: true
I20251023 20:15:17.072220 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 4f47ddde9f3d4042bc84853332e21950 (DEFAULT_TABLE table=default.Table_888857831b2f4735a4d944a2318eae73 [id=651489d40b6b425897b71f97e89c6dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.072616 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f47ddde9f3d4042bc84853332e21950. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.075588 23783 ts_tablet_manager.cc:1929] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.075724 23783 log.cc:1199] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/091e54a94f2d4c40a0006c7e35f74823
I20251023 20:15:17.076085 23783 ts_tablet_manager.cc:1950] T 091e54a94f2d4c40a0006c7e35f74823 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.077020 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 091e54a94f2d4c40a0006c7e35f74823 (table default.Table_8e8a728158404934a7b077db56620986 [id=a7ccdb7723b44991be2809bf1d80d6a6]) successfully deleted
I20251023 20:15:17.080629 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.080942 23785 tablet_replica.cc:333] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:17.081187 23785 raft_consensus.cc:2243] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.081441 23785 raft_consensus.cc:2272] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.082235 23785 ts_tablet_manager.cc:1916] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.085680 23783 tablet_replica.cc:333] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:17.087440 23134 consensus_peers.cc:597] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 -> Peer 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331): Couldn't send request to peer 6545140d94b7485aa81c51ad233355cd. Status: Network error: Client connection negotiation failed: client connection to 127.22.127.190:34331: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251023 20:15:17.087795 23783 raft_consensus.cc:2243] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.088234 23783 raft_consensus.cc:2272] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.088709 23785 ts_tablet_manager.cc:1929] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.088815 23785 log.cc:1199] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/799132aedc784de3bf0fc6478c8ef496
I20251023 20:15:17.089361 23783 ts_tablet_manager.cc:1916] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.089440 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 (DEFAULT_TABLE table=default.Table_9a30e9c9272f47eb8fccd5e48ce47c30 [id=8b9eb030f1a24f68bb8e2e87b580727a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.090617 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c317dc31cb0448d985512e6ba98735be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.091331 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e3d422a66f432789c0cf4a00e57af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.093508 23502 tablet_service.cc:1505] Processing CreateTablet for tablet 4f47ddde9f3d4042bc84853332e21950 (DEFAULT_TABLE table=default.Table_888857831b2f4735a4d944a2318eae73 [id=651489d40b6b425897b71f97e89c6dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.093838 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 (DEFAULT_TABLE table=default.Table_9a30e9c9272f47eb8fccd5e48ce47c30 [id=8b9eb030f1a24f68bb8e2e87b580727a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.094030 23509 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.094568 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fad48a232deb4365956d805e803043b0" } new_table_name: "default.Table_21f0d4fca62a4309816edd74c6b137bf" modify_external_catalogs: true
I20251023 20:15:17.096012 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_94257ba5c56c4269842dcd5dd0d19461" } new_table_name: "default.Table_ca4eab8d09574f7abb17e56482816adf" modify_external_catalogs: true
I20251023 20:15:17.098275 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21ec49194e24723b31fd9c86f1f85e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.098773 23500 tablet_service.cc:1505] Processing CreateTablet for tablet 2532c8567db14901b383f42e5f04b4c2 (DEFAULT_TABLE table=default.Table_bf99025eb142454fa0513fafcbe636aa [id=f52f1d83bd9345ffa08e6d3d282914e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.099215 23790 tablet_bootstrap.cc:492] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.099215 23787 tablet_bootstrap.cc:492] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.099362 23790 ts_tablet_manager.cc:1403] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.004s
I20251023 20:15:17.099408 23787 ts_tablet_manager.cc:1403] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.005s
I20251023 20:15:17.100090 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e3d422a66f432789c0cf4a00e57af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.100219 23790 raft_consensus.cc:359] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.100266 23787 raft_consensus.cc:359] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.100409 23790 raft_consensus.cc:385] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.100490 23790 raft_consensus.cc:740] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.100514 23787 raft_consensus.cc:385] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.100627 23500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2532c8567db14901b383f42e5f04b4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.100661 23787 raft_consensus.cc:740] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.100939 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f47ddde9f3d4042bc84853332e21950. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.100805 23790 consensus_queue.cc:260] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.100970 23787 consensus_queue.cc:260] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.103655 23787 ts_tablet_manager.cc:1434] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:17.103772 23790 ts_tablet_manager.cc:1434] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251023 20:15:17.104023 23787 tablet_bootstrap.cc:492] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.104799 23790 tablet_bootstrap.cc:492] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.104846 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 2177416169ca4d0d8368b1cac90c513e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:14 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.105674 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d73da547fd04557abd5aba73e76dd18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.107831 23785 ts_tablet_manager.cc:1950] T 799132aedc784de3bf0fc6478c8ef496 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:17.108268 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95574277dde44983bb0a43d2b5ad01d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.108831 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 799132aedc784de3bf0fc6478c8ef496 (table default.Table_a90cbb8972924d6aad10abd0327c9874 [id=b2d8f2d64c7e4668a8e274bcdc384754]) successfully deleted
I20251023 20:15:17.108961 23787 tablet_bootstrap.cc:654] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.109030 23785 tablet_replica.cc:333] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:17.109218 23785 raft_consensus.cc:2243] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.109400 23785 raft_consensus.cc:2272] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:17.109445 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2532c8567db14901b383f42e5f04b4c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2532c8567db14901b383f42e5f04b4c2 already in progress: creating tablet
W20251023 20:15:17.109704 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9d1ddde7e3134e23b3a9356523a93753 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d1ddde7e3134e23b3a9356523a93753 already in progress: creating tablet
I20251023 20:15:17.110586 23790 tablet_bootstrap.cc:654] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.111711 23785 ts_tablet_manager.cc:1916] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.122246 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet e0f8d8184a68453abd2792ccc7740aa1 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.122074 23783 ts_tablet_manager.cc:1929] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.123633 23783 log.cc:1199] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/24528de9d0a84eddb091dec4f1d2b4e2
I20251023 20:15:17.124037 23783 ts_tablet_manager.cc:1950] T 24528de9d0a84eddb091dec4f1d2b4e2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.126312 23787 tablet_bootstrap.cc:492] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.126574 23787 ts_tablet_manager.cc:1403] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251023 20:15:17.127292 23785 ts_tablet_manager.cc:1929] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.127451 23785 log.cc:1199] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/0d73da547fd04557abd5aba73e76dd18
I20251023 20:15:17.127485 23787 raft_consensus.cc:359] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.127805 23785 ts_tablet_manager.cc:1950] T 0d73da547fd04557abd5aba73e76dd18 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:17.127928 23787 raft_consensus.cc:385] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.128039 23787 raft_consensus.cc:740] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.128391 23787 consensus_queue.cc:260] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.130802 23787 ts_tablet_manager.cc:1434] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:17.131465 23787 tablet_bootstrap.cc:492] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.131541 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 0d73da547fd04557abd5aba73e76dd18 (table default.Table_d592564ff9cd44ab86010457642460ec [id=8d6a2964df664438b8680d81e4f040aa]) successfully deleted
W20251023 20:15:17.133659 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 4beebee06e5341a29c65d0c3cf985ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4beebee06e5341a29c65d0c3cf985ef4 already in progress: creating tablet
W20251023 20:15:17.134748 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet bd88285bf0a94fad858a25004609131c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd88285bf0a94fad858a25004609131c already in progress: creating tablet
I20251023 20:15:17.135109 23790 tablet_bootstrap.cc:492] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.135286 23790 ts_tablet_manager.cc:1403] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.009s
I20251023 20:15:17.135665 23787 tablet_bootstrap.cc:654] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:17.136055 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3894dd4feeea4c16861a6c9520059474 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3894dd4feeea4c16861a6c9520059474 already in progress: creating tablet
W20251023 20:15:17.136466 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 451955aee3aa4bcb82252bcefbbcaefe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 451955aee3aa4bcb82252bcefbbcaefe already in progress: creating tablet
W20251023 20:15:17.136732 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet edec01a63a3046d9ae5a372fc9068b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edec01a63a3046d9ae5a372fc9068b52 already in progress: creating tablet
W20251023 20:15:17.137072 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ef1229b9e0384a82811f244d2bf4c8ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef1229b9e0384a82811f244d2bf4c8ce already in progress: creating tablet
I20251023 20:15:17.137593 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 799132aedc784de3bf0fc6478c8ef496 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.138126 23790 raft_consensus.cc:359] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.138337 23790 raft_consensus.cc:385] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.138424 23790 raft_consensus.cc:740] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.138850 23790 consensus_queue.cc:260] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.139310 23783 tablet_replica.cc:333] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.139530 23783 raft_consensus.cc:2243] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.139747 23783 raft_consensus.cc:2272] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.140506 23783 ts_tablet_manager.cc:1916] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.141387 23790 ts_tablet_manager.cc:1434] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251023 20:15:17.142822 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet c3035573f53c4a6db0bb72882ca84f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.143334 23785 tablet_replica.cc:333] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:17.143484 23785 raft_consensus.cc:2243] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.143669 23785 raft_consensus.cc:2272] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.145233 23785 ts_tablet_manager.cc:1916] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.146296 23787 tablet_bootstrap.cc:492] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.146484 23787 ts_tablet_manager.cc:1403] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20251023 20:15:17.147251 23787 raft_consensus.cc:359] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.147485 23787 raft_consensus.cc:385] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.147596 23787 raft_consensus.cc:740] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.147873 23787 consensus_queue.cc:260] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.148017 23790 tablet_bootstrap.cc:492] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.149518 23787 ts_tablet_manager.cc:1434] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:17.149848 23787 tablet_bootstrap.cc:492] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.150219 23790 tablet_bootstrap.cc:654] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.153338 23787 tablet_bootstrap.cc:654] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.154281 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 955412de8bd1482ea7304872b48f38c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 283166bbd8004ce3bca9f2ecbdbcbf48) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.155285 23785 ts_tablet_manager.cc:1929] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.155412 23785 log.cc:1199] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/c3035573f53c4a6db0bb72882ca84f37
I20251023 20:15:17.155795 23785 ts_tablet_manager.cc:1950] T c3035573f53c4a6db0bb72882ca84f37 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:17.156560 23783 ts_tablet_manager.cc:1929] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.156725 23783 log.cc:1199] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d8ad21e3888a4915ae5f252c2fca64cb
I20251023 20:15:17.156800 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet c3035573f53c4a6db0bb72882ca84f37 (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:17.157215 23783 ts_tablet_manager.cc:1950] T d8ad21e3888a4915ae5f252c2fca64cb P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.158437 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d8ad21e3888a4915ae5f252c2fca64cb (table default.Table_57a2a81ca5484113b5e2a0857d40dbbb [id=fc58c27a539f4454acbba8fbf3b62461]) successfully deleted
I20251023 20:15:17.158820 23783 tablet_replica.cc:333] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.159063 23783 raft_consensus.cc:2243] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.159272 23783 raft_consensus.cc:2272] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.160084 23783 ts_tablet_manager.cc:1916] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.165261 23790 tablet_bootstrap.cc:492] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.165426 23790 ts_tablet_manager.cc:1403] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20251023 20:15:17.165756 23787 tablet_bootstrap.cc:492] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.165966 23787 ts_tablet_manager.cc:1403] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20251023 20:15:17.166263 23790 raft_consensus.cc:359] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.166463 23790 raft_consensus.cc:385] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.166567 23790 raft_consensus.cc:740] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.166894 23787 raft_consensus.cc:359] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.166841 23790 consensus_queue.cc:260] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.167204 23787 raft_consensus.cc:385] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.167323 23787 raft_consensus.cc:740] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.167603 23787 consensus_queue.cc:260] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.168990 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet c3035573f53c4a6db0bb72882ca84f37 with delete_type TABLET_DATA_DELETED (replaced by tablet 73b4abd308054bac9d49b1d8e7dd2ebc) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.169556 23787 ts_tablet_manager.cc:1434] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:17.169836 23787 tablet_bootstrap.cc:492] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:17.171015 23134 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c3035573f53c4a6db0bb72882ca84f37 because the tablet was not found. No further retry: Not found: Tablet not found: c3035573f53c4a6db0bb72882ca84f37
I20251023 20:15:17.171228 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet dc8f615c621043da88d52f3964b0a0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.171530 23790 ts_tablet_manager.cc:1434] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:17.173228 23787 tablet_bootstrap.cc:654] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.173568 23790 tablet_bootstrap.cc:492] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.175097 23790 tablet_bootstrap.cc:654] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.183035 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8ac87d9615a74c5c858be1d6c31772a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.183655 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac87d9615a74c5c858be1d6c31772a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.190380 23784 ts_tablet_manager.cc:1929] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.190533 23784 log.cc:1199] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/bd88285bf0a94fad858a25004609131c
I20251023 20:15:17.190912 23784 ts_tablet_manager.cc:1950] T bd88285bf0a94fad858a25004609131c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:17.192870 23783 ts_tablet_manager.cc:1929] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.193496 23783 log.cc:1199] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/216e2c887a364c8ea5068c3834d1e4e2
I20251023 20:15:17.194010 23783 ts_tablet_manager.cc:1950] T 216e2c887a364c8ea5068c3834d1e4e2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.198329 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 216e2c887a364c8ea5068c3834d1e4e2 (table default.Table_d94baddda6104b5c9c007c95623ea0dd [id=ce02fa77823e4fbcad36a7e21bf10630]) successfully deleted
W20251023 20:15:17.200078 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 95574277dde44983bb0a43d2b5ad01d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95574277dde44983bb0a43d2b5ad01d5 already in progress: creating tablet
W20251023 20:15:17.201237 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9d1ddde7e3134e23b3a9356523a93753 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d1ddde7e3134e23b3a9356523a93753 already in progress: creating tablet
W20251023 20:15:17.202536 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2532c8567db14901b383f42e5f04b4c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2532c8567db14901b383f42e5f04b4c2 already in progress: creating tablet
I20251023 20:15:17.203218 23783 tablet_replica.cc:333] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.203662 23783 raft_consensus.cc:2243] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.204627 23783 raft_consensus.cc:2272] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.204416 23787 tablet_bootstrap.cc:492] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.204898 23787 ts_tablet_manager.cc:1403] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.005s
I20251023 20:15:17.205832 23787 raft_consensus.cc:359] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.206096 23787 raft_consensus.cc:385] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.206216 23787 raft_consensus.cc:740] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.206511 23787 consensus_queue.cc:260] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.208750 23787 ts_tablet_manager.cc:1434] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:17.209429 23787 tablet_bootstrap.cc:492] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.210909 23786 tablet_bootstrap.cc:492] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:17.211066 23786 ts_tablet_manager.cc:1403] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.313s user 0.005s sys 0.006s
I20251023 20:15:17.211922 23786 raft_consensus.cc:359] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.212118 23786 raft_consensus.cc:385] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.212215 23786 raft_consensus.cc:740] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.212350 23783 ts_tablet_manager.cc:1916] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.212451 23786 consensus_queue.cc:260] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.218539 23787 tablet_bootstrap.cc:654] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.223796 23790 tablet_bootstrap.cc:492] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.224011 23790 ts_tablet_manager.cc:1403] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.050s user 0.020s sys 0.023s
I20251023 20:15:17.224781 23790 raft_consensus.cc:359] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.224992 23790 raft_consensus.cc:385] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.225066 23790 raft_consensus.cc:740] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.225366 23790 consensus_queue.cc:260] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.227161 23783 ts_tablet_manager.cc:1929] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:17.227962 23827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:17.228384 23827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:17.228567 23827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251023 20:15:17.231294 23790 ts_tablet_manager.cc:1434] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251023 20:15:17.227967 23787 tablet_bootstrap.cc:492] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.232483 23790 tablet_bootstrap.cc:492] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.232697 23787 ts_tablet_manager.cc:1403] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20251023 20:15:17.233660 23787 raft_consensus.cc:359] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.233916 23787 raft_consensus.cc:385] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.234026 23787 raft_consensus.cc:740] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.234138 23790 tablet_bootstrap.cc:654] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.234397 23787 consensus_queue.cc:260] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.235864 23787 ts_tablet_manager.cc:1434] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:17.236554 23787 tablet_bootstrap.cc:492] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.238600 23783 log.cc:1199] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/2177416169ca4d0d8368b1cac90c513e
I20251023 20:15:17.239116 23787 tablet_bootstrap.cc:654] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.239326 23783 ts_tablet_manager.cc:1950] T 2177416169ca4d0d8368b1cac90c513e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.241194 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 2177416169ca4d0d8368b1cac90c513e (table default.Table_b17a9a8073ae4fe689a0ba8d2f0cead3 [id=2d40a1f71ab847df9127c80fd25f05af]) successfully deleted
I20251023 20:15:17.241569 23783 tablet_replica.cc:333] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.241755 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet bd88285bf0a94fad858a25004609131c (table default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]) successfully deleted
I20251023 20:15:17.241828 23783 raft_consensus.cc:2243] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.241978 23783 raft_consensus.cc:2272] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:17.242281 23827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:17.242415 23827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:17.242409 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 451955aee3aa4bcb82252bcefbbcaefe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 451955aee3aa4bcb82252bcefbbcaefe already in progress: creating tablet
W20251023 20:15:17.242605 23827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:17.242715 23827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:17.242760 23783 ts_tablet_manager.cc:1916] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:17.242973 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3894dd4feeea4c16861a6c9520059474 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3894dd4feeea4c16861a6c9520059474 already in progress: creating tablet
W20251023 20:15:17.243463 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet edec01a63a3046d9ae5a372fc9068b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edec01a63a3046d9ae5a372fc9068b52 already in progress: creating tablet
I20251023 20:15:17.243505 23790 tablet_bootstrap.cc:492] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.243654 23790 ts_tablet_manager.cc:1403] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
W20251023 20:15:17.243975 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ef1229b9e0384a82811f244d2bf4c8ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef1229b9e0384a82811f244d2bf4c8ce already in progress: creating tablet
I20251023 20:15:17.244370 23790 raft_consensus.cc:359] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.244553 23790 raft_consensus.cc:385] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.244642 23790 raft_consensus.cc:740] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.244890 23790 consensus_queue.cc:260] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.246413 23790 ts_tablet_manager.cc:1434] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20251023 20:15:17.246469 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9d1ddde7e3134e23b3a9356523a93753 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d1ddde7e3134e23b3a9356523a93753 already in progress: creating tablet
I20251023 20:15:17.247462 23790 tablet_bootstrap.cc:492] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:17.247471 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 95574277dde44983bb0a43d2b5ad01d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95574277dde44983bb0a43d2b5ad01d5 already in progress: creating tablet
I20251023 20:15:17.248477 23810 raft_consensus.cc:493] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.248739 23810 raft_consensus.cc:515] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.250476 23810 leader_election.cc:290] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:17.250734 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2532c8567db14901b383f42e5f04b4c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2532c8567db14901b383f42e5f04b4c2 already in progress: creating tablet
I20251023 20:15:17.252640 23783 ts_tablet_manager.cc:1929] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.252830 23783 log.cc:1199] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/799132aedc784de3bf0fc6478c8ef496
I20251023 20:15:17.253228 23783 ts_tablet_manager.cc:1950] T 799132aedc784de3bf0fc6478c8ef496 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.254238 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 799132aedc784de3bf0fc6478c8ef496 (table default.Table_a90cbb8972924d6aad10abd0327c9874 [id=b2d8f2d64c7e4668a8e274bcdc384754]) successfully deleted
I20251023 20:15:17.254695 23783 tablet_replica.cc:333] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.254922 23783 raft_consensus.cc:2243] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.255146 23783 raft_consensus.cc:2272] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.255903 23783 ts_tablet_manager.cc:1916] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.262354 23783 ts_tablet_manager.cc:1929] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.262590 23783 log.cc:1199] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/955412de8bd1482ea7304872b48f38c0
I20251023 20:15:17.263013 23783 ts_tablet_manager.cc:1950] T 955412de8bd1482ea7304872b48f38c0 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.263263 23827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.190:34331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.127.190
--webserver_port=44199
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
I20251023 20:15:17.264359 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 955412de8bd1482ea7304872b48f38c0 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:17.264804 23783 tablet_replica.cc:333] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.265049 23783 raft_consensus.cc:2243] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.265276 23783 raft_consensus.cc:2272] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.265913 23786 ts_tablet_manager.cc:1434] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20251023 20:15:17.266111 23783 ts_tablet_manager.cc:1916] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.266249 23786 tablet_bootstrap.cc:492] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:17.267859 23827 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:17.270565 23827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:15:17.272475 23783 ts_tablet_manager.cc:1929] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.272696 23783 log.cc:1199] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/dc8f615c621043da88d52f3964b0a0ec
I20251023 20:15:17.273065 23783 ts_tablet_manager.cc:1950] T dc8f615c621043da88d52f3964b0a0ec P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.274039 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet dc8f615c621043da88d52f3964b0a0ec (table default.Table_4fba73266b6d4d36958e8f0566eb624e [id=d7983b16809245a2a73b2a6cd79f6e29]) successfully deleted
I20251023 20:15:17.276283 23787 tablet_bootstrap.cc:492] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.276417 23787 ts_tablet_manager.cc:1403] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.001s
I20251023 20:15:17.277153 23787 raft_consensus.cc:359] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.277340 23787 raft_consensus.cc:385] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.277431 23787 raft_consensus.cc:740] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.277639 23787 consensus_queue.cc:260] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.278127 23787 ts_tablet_manager.cc:1434] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:17.278442 23787 tablet_bootstrap.cc:492] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.282754 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:17.284337 23787 tablet_bootstrap.cc:654] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.285117 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.285584 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:17.285844 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f already in progress: creating tablet
I20251023 20:15:17.286980 23790 tablet_bootstrap.cc:654] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:17.288524 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f already in progress: creating tablet
I20251023 20:15:17.290493 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet cc5ce157b6ca42da90b051a184983bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.291934 23786 tablet_bootstrap.cc:654] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.292529 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:17.293208 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f already in progress: creating tablet
I20251023 20:15:17.297094 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet bd88285bf0a94fad858a25004609131c with delete_type TABLET_DATA_DELETED (replaced by tablet c317dc31cb0448d985512e6ba98735be) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.297214 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 52bc660b9da54c5d984381a267e1efa2
I20251023 20:15:17.297827 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.297895 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_cdf01488bd3e4c7c98f16522764f7a18" } new_table_name: "default.Table_d07400a150ac4a6c86bac1caeebe8739" modify_external_catalogs: true
I20251023 20:15:17.297947 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_441bf8050be9453c85aef2cb7ac63ea4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.298410 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_441bf8050be9453c85aef2cb7ac63ea4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.298470 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d2b87a7247624b0380ee21ec002b9f77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.298527 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_1fc3a0e0fe484168b2f767fdcc3eba64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.298960 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fc3a0e0fe484168b2f767fdcc3eba64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.300139 23185 master_runner.cc:186] Error getting master registration for 127.22.127.190:34331: Network error: Client connection negotiation failed: client connection to 127.22.127.190:34331: connect: Connection refused (error 111),
I20251023 20:15:17.300287 23185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251023 20:15:17.298959 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2b87a7247624b0380ee21ec002b9f77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.303377 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.303789 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.303989 23783 tablet_replica.cc:333] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.304394 23783 raft_consensus.cc:2243] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:17.304687 23333 leader_election.cc:343] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: d8ad21e3888a4915ae5f252c2fca64cb
I20251023 20:15:17.304852 23783 raft_consensus.cc:2272] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:17.305202 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 52bc660b9da54c5d984381a267e1efa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52bc660b9da54c5d984381a267e1efa2 already in progress: creating tablet
I20251023 20:15:17.306295 23783 ts_tablet_manager.cc:1916] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.305228 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 955412de8bd1482ea7304872b48f38c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 283166bbd8004ce3bca9f2ecbdbcbf48) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:17.310863 23333 leader_election.cc:343] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d8ad21e3888a4915ae5f252c2fca64cb
I20251023 20:15:17.311046 23333 leader_election.cc:304] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:17.311488 23810 raft_consensus.cc:2749] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:17.312642 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 52bc660b9da54c5d984381a267e1efa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52bc660b9da54c5d984381a267e1efa2 already in progress: creating tablet
I20251023 20:15:17.313041 23784 tablet_replica.cc:333] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:17.313293 23784 raft_consensus.cc:2243] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.313481 23784 raft_consensus.cc:2272] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.314214 23790 tablet_bootstrap.cc:492] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.314291 23784 ts_tablet_manager.cc:1916] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.314486 23790 ts_tablet_manager.cc:1403] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.002s
I20251023 20:15:17.315570 23790 raft_consensus.cc:359] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.315824 23790 raft_consensus.cc:385] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.315929 23790 raft_consensus.cc:740] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.316308 23790 consensus_queue.cc:260] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.316730 23810 raft_consensus.cc:493] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.317157 23810 raft_consensus.cc:515] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.317833 23810 leader_election.cc:290] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:17.317814 23783 ts_tablet_manager.cc:1929] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.317967 23783 log.cc:1199] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/cc5ce157b6ca42da90b051a184983bfe
W20251023 20:15:17.319027 23333 leader_election.cc:343] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: cc5ce157b6ca42da90b051a184983bfe
I20251023 20:15:17.320043 23783 ts_tablet_manager.cc:1950] T cc5ce157b6ca42da90b051a184983bfe P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.320683 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet cc5ce157b6ca42da90b051a184983bfe (table default.Table_bd046e3e4fd9439c8aabbcee605207e4 [id=8b2cfa043c8147098454bfead6809936]) successfully deleted
I20251023 20:15:17.321054 23783 tablet_replica.cc:333] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.319473 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc5ce157b6ca42da90b051a184983bfe" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:17.319664 23787 tablet_bootstrap.cc:492] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.321660 23787 ts_tablet_manager.cc:1403] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.002s
I20251023 20:15:17.322393 23787 raft_consensus.cc:359] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.322569 23787 raft_consensus.cc:385] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.322647 23787 raft_consensus.cc:740] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.322903 23787 consensus_queue.cc:260] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:17.323436 23333 leader_election.cc:343] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: Tablet not RUNNING: SHUTDOWN
I20251023 20:15:17.323664 23333 leader_election.cc:304] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:17.323570 23790 ts_tablet_manager.cc:1434] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251023 20:15:17.324384 23787 ts_tablet_manager.cc:1434] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:17.324976 23787 tablet_bootstrap.cc:492] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.326206 23790 tablet_bootstrap.cc:492] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.326364 23810 raft_consensus.cc:2749] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:17.326692 23787 tablet_bootstrap.cc:654] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.327553 23783 raft_consensus.cc:2243] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.327780 23783 raft_consensus.cc:2272] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.328778 23783 ts_tablet_manager.cc:1916] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.329195 23790 tablet_bootstrap.cc:654] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.335155 23783 ts_tablet_manager.cc:1929] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:17.341331 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf has no consensus state
I20251023 20:15:17.342475 23786 tablet_bootstrap.cc:492] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:17.342670 23786 ts_tablet_manager.cc:1403] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.002s
I20251023 20:15:17.342693 23787 tablet_bootstrap.cc:492] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.343549 23786 raft_consensus.cc:359] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.343761 23786 raft_consensus.cc:385] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.343847 23786 raft_consensus.cc:740] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.344115 23786 consensus_queue.cc:260] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.342875 23787 ts_tablet_manager.cc:1403] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20251023 20:15:17.345686 23787 raft_consensus.cc:359] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.345870 23787 raft_consensus.cc:385] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.345968 23787 raft_consensus.cc:740] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.346184 23787 consensus_queue.cc:260] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.346704 23784 ts_tablet_manager.cc:1929] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.347801 23787 ts_tablet_manager.cc:1434] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:17.349316 23787 tablet_bootstrap.cc:492] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:17.350059 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cff337a6e728489b99fa8c5d4dfece2b consensus state has no leader
I20251023 20:15:17.351047 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_62d382e981754b839d2c87efc8004825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.351344 23783 log.cc:1199] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/bd88285bf0a94fad858a25004609131c
I20251023 20:15:17.351567 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_59bd96856079400e8eda1b518a1ec457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.351676 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62d382e981754b839d2c87efc8004825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.351729 23787 tablet_bootstrap.cc:654] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:17.352046 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59bd96856079400e8eda1b518a1ec457 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.352633 23783 ts_tablet_manager.cc:1950] T bd88285bf0a94fad858a25004609131c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:17.353268 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cec51c54ea534b1da4b1d170e79d9f1c consensus state has no leader
I20251023 20:15:17.353410 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_191b94017cca4cfb946f38ab19ebfe20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.353628 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_5dbd1e92b4384903b369416019cf70b9" } new_table_name: "default.Table_af01354e99f74972ac276e46643f9dba" modify_external_catalogs: true
W20251023 20:15:17.353904 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_191b94017cca4cfb946f38ab19ebfe20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.355319 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet bd88285bf0a94fad858a25004609131c (table default.Table_4f168f47af364b3db52fbbc64de90a6e [id=8d6a76224aec482db23a4b660e13a98c]) successfully deleted
W20251023 20:15:17.355597 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 52bc660b9da54c5d984381a267e1efa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52bc660b9da54c5d984381a267e1efa2 already in progress: creating tablet
I20251023 20:15:17.356034 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet e0f8d8184a68453abd2792ccc7740aa1
I20251023 20:15:17.356042 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685213b588148febd6a23e1d2d6691d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.356269 23783 tablet_replica.cc:333] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.356418 23783 raft_consensus.cc:2243] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.356526 23140 catalog_manager.cc:6719] ReplaceTablet: tablet e0f8d8184a68453abd2792ccc7740aa1 deleted and replaced by tablet cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:17.356581 23783 raft_consensus.cc:2272] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.356843 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8d8ada20a7bb43eb9a534aedca23cfd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.357321 23783 ts_tablet_manager.cc:1916] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:17.358335 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d8ada20a7bb43eb9a534aedca23cfd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.358361 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.359071 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.361672 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 2e338b982a564ea5b1b15c2146a96fa2 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.362146 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 50c75cce4b9d4c48800b2d0908676a14 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.362692 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e13d2545a0a54b91a8bcb64db45581cc" } modify_external_catalogs: true
I20251023 20:15:17.362763 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 with delete_type TABLET_DATA_DELETED (replaced by tablet e5368813164949ff80e27bf59dcc931a) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:17.362767 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e0f8d8184a68453abd2792ccc7740aa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f8d8184a68453abd2792ccc7740aa1 already in progress: creating tablet
I20251023 20:15:17.357277 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e31aebfc87734ffeb2c0b25c07cb3f70) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.362862 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e13d2545a0a54b91a8bcb64db45581cc" } modify_external_catalogs: true
I20251023 20:15:17.362905 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 8289ce37fa074891beb584bf2fff9747 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.362391 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.363111 23784 log.cc:1199] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/955412de8bd1482ea7304872b48f38c0
I20251023 20:15:17.364677 23786 ts_tablet_manager.cc:1434] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251023 20:15:17.368304 23787 tablet_bootstrap.cc:492] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.368351 23784 ts_tablet_manager.cc:1950] T 955412de8bd1482ea7304872b48f38c0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:17.368463 23787 ts_tablet_manager.cc:1403] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20251023 20:15:17.369176 23786 tablet_bootstrap.cc:492] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:17.369270 23787 raft_consensus.cc:359] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.369446 23787 raft_consensus.cc:385] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.369516 23787 raft_consensus.cc:740] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.369761 23787 consensus_queue.cc:260] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.372011 23787 ts_tablet_manager.cc:1434] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:17.372103 23786 tablet_bootstrap.cc:654] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.372315 23787 tablet_bootstrap.cc:492] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.373927 23787 tablet_bootstrap.cc:654] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.374368 23790 tablet_bootstrap.cc:492] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.374563 23790 ts_tablet_manager.cc:1403] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.004s
W20251023 20:15:17.375083 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe922c438a894a7aa04699b2d2897a28 has no consensus state
I20251023 20:15:17.375375 23790 raft_consensus.cc:359] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.375522 23783 ts_tablet_manager.cc:1929] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.375563 23790 raft_consensus.cc:385] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.375644 23783 log.cc:1199] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/1685213b588148febd6a23e1d2d6691d
I20251023 20:15:17.375643 23790 raft_consensus.cc:740] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.376015 23783 ts_tablet_manager.cc:1950] T 1685213b588148febd6a23e1d2d6691d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.375936 23790 consensus_queue.cc:260] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.376951 23783 tablet_replica.cc:333] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.377168 23783 raft_consensus.cc:2243] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.377363 23783 raft_consensus.cc:2272] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.378160 23783 ts_tablet_manager.cc:1916] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.378273 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 1685213b588148febd6a23e1d2d6691d (table default.Table_2e1bf7e58ab84283b5f61b77865778ca [id=580a286a829e44f9b9992f3b0b7482eb]) successfully deleted
W20251023 20:15:17.378656 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e0f8d8184a68453abd2792ccc7740aa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f8d8184a68453abd2792ccc7740aa1 already in progress: creating tablet
I20251023 20:15:17.378870 23790 ts_tablet_manager.cc:1434] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20251023 20:15:17.378820 23920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:17.379896 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0197c34b4744436b8ba95be1ba33be9f" } modify_external_catalogs: true
I20251023 20:15:17.380229 23790 tablet_bootstrap.cc:492] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.380194 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0197c34b4744436b8ba95be1ba33be9f" } modify_external_catalogs: true
I20251023 20:15:17.381130 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 5416da30c66645c19d36112980608527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.381654 23787 tablet_bootstrap.cc:492] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.381862 23787 ts_tablet_manager.cc:1403] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
W20251023 20:15:17.382195 23924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:17.382390 23790 tablet_bootstrap.cc:654] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.382609 23787 raft_consensus.cc:359] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.382790 23787 raft_consensus.cc:385] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.382876 23787 raft_consensus.cc:740] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.383121 23787 consensus_queue.cc:260] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.383734 23787 ts_tablet_manager.cc:1434] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:17.384030 23787 tablet_bootstrap.cc:492] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.386370 23787 tablet_bootstrap.cc:654] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.387796 23784 tablet_replica.cc:333] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:17.391105 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 955412de8bd1482ea7304872b48f38c0 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
W20251023 20:15:17.394364 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e consensus state has no leader
I20251023 20:15:17.394675 23145 master_service.cc:954] ReplaceTablet: received request to replace tablet ef1229b9e0384a82811f244d2bf4c8ce from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.394773 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4740fc3fe4ac4927b7f776c980467298" } modify_external_catalogs: true
I20251023 20:15:17.394966 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4740fc3fe4ac4927b7f776c980467298" } modify_external_catalogs: true
I20251023 20:15:17.396121 23784 raft_consensus.cc:2243] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.396426 23784 raft_consensus.cc:2272] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.398213 23786 tablet_bootstrap.cc:492] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:17.398448 23786 ts_tablet_manager.cc:1403] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20251023 20:15:17.399286 23786 raft_consensus.cc:359] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.399485 23786 raft_consensus.cc:385] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.399580 23786 raft_consensus.cc:740] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.399852 23786 consensus_queue.cc:260] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.400344 23786 ts_tablet_manager.cc:1434] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:17.401695 23786 tablet_bootstrap.cc:492] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:17.403278 23786 tablet_bootstrap.cc:654] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.404817 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.405260 23784 ts_tablet_manager.cc:1916] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.408074 23790 tablet_bootstrap.cc:492] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.408253 23790 ts_tablet_manager.cc:1403] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20251023 20:15:17.408337 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 949e31d4a9ee4579843ed54153e22c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 9c0aeac2f20c4ca48c130a0c851f290f) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.409294 23790 raft_consensus.cc:359] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.409502 23790 raft_consensus.cc:385] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.409582 23790 raft_consensus.cc:740] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.409817 23790 consensus_queue.cc:260] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.410298 23790 ts_tablet_manager.cc:1434] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:17.410595 23790 tablet_bootstrap.cc:492] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.412827 23790 tablet_bootstrap.cc:654] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.422636 23146 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998]). Expected version 4 got 0
I20251023 20:15:17.426498 23377 tablet_service.cc:1505] Processing CreateTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 (DEFAULT_TABLE table=default.Table_ea99b2e89c224754a8eba8255433cd14 [id=a431259b5e254da6941241ffcf8ce597]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.426968 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa794b3bec4b4b3993c814ea7cd02ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.427855 23784 ts_tablet_manager.cc:1929] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.427990 23784 log.cc:1199] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/02b79ba69a1c4f8ebc7ecb795ac0d1d5
I20251023 20:15:17.428385 23784 ts_tablet_manager.cc:1950] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:17.428838 23501 tablet_service.cc:1505] Processing CreateTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 (DEFAULT_TABLE table=default.Table_ea99b2e89c224754a8eba8255433cd14 [id=a431259b5e254da6941241ffcf8ce597]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.429332 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf (DEFAULT_TABLE table=default.Table_1fd972f3964245fab674e2cfc26a8210 [id=cf5e426caddf4508bc0780b9809fff5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.429661 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.429602 23637 tablet_service.cc:1505] Processing CreateTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 (DEFAULT_TABLE table=default.Table_ea99b2e89c224754a8eba8255433cd14 [id=a431259b5e254da6941241ffcf8ce597]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.429842 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 5346a221c78342359eb6e418bd12cd24 (DEFAULT_TABLE table=default.Table_914e9b5e4a7443c5baf677b4b6857359 [id=71ba16e8b75642e09686eb46c4315366]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.429986 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa794b3bec4b4b3993c814ea7cd02ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.430174 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5346a221c78342359eb6e418bd12cd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.430270 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 5346a221c78342359eb6e418bd12cd24 (DEFAULT_TABLE table=default.Table_914e9b5e4a7443c5baf677b4b6857359 [id=71ba16e8b75642e09686eb46c4315366]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.430594 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa794b3bec4b4b3993c814ea7cd02ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.429610 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf (DEFAULT_TABLE table=default.Table_1fd972f3964245fab674e2cfc26a8210 [id=cf5e426caddf4508bc0780b9809fff5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.431603 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.433988 23783 ts_tablet_manager.cc:1929] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.434120 23783 log.cc:1199] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e44dfb4fd3d04a58b5fcc6102e055ff1
I20251023 20:15:17.434482 23783 ts_tablet_manager.cc:1950] T e44dfb4fd3d04a58b5fcc6102e055ff1 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.436309 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5346a221c78342359eb6e418bd12cd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.439532 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 (DEFAULT_TABLE table=default.Table_c27155507a7f45cc860ea6089419e20e [id=4bbc0f95fee64c7c8570961ba3f1c0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.439895 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fa6a838b1bd409ba2a4e2074576ee23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.440043 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 (DEFAULT_TABLE table=default.Table_c27155507a7f45cc860ea6089419e20e [id=4bbc0f95fee64c7c8570961ba3f1c0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.440400 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fa6a838b1bd409ba2a4e2074576ee23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.441870 23503 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 (DEFAULT_TABLE table=default.Table_e926815d67914fed915b184838f05b72 [id=265a9707ddea4ee1a19aac6033883822]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.442198 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.445288 23644 tablet_service.cc:1505] Processing CreateTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c (DEFAULT_TABLE table=default.Table_4e18a968658f470d870bc932102ebf00 [id=ad634b38921149658d6212b02c421514]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.445438 23508 tablet_service.cc:1505] Processing CreateTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c (DEFAULT_TABLE table=default.Table_4e18a968658f470d870bc932102ebf00 [id=ad634b38921149658d6212b02c421514]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.445636 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ba7ab28dfa4ac38e0b58ba2115495c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.445763 23783 tablet_replica.cc:333] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.445827 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ba7ab28dfa4ac38e0b58ba2115495c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.445935 23783 raft_consensus.cc:2243] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.446120 23783 raft_consensus.cc:2272] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.447983 23783 ts_tablet_manager.cc:1916] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.448370 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet e44dfb4fd3d04a58b5fcc6102e055ff1 (table default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:17.449726 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 (DEFAULT_TABLE table=default.Table_e926815d67914fed915b184838f05b72 [id=265a9707ddea4ee1a19aac6033883822]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.450090 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.450706 23515 tablet_service.cc:1505] Processing CreateTablet for tablet fe922c438a894a7aa04699b2d2897a28 (DEFAULT_TABLE table=default.Table_d07400a150ac4a6c86bac1caeebe8739 [id=fe2df9a958ec4863ab2100750ba7dc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.451071 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe922c438a894a7aa04699b2d2897a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.451355 23650 tablet_service.cc:1505] Processing CreateTablet for tablet fe922c438a894a7aa04699b2d2897a28 (DEFAULT_TABLE table=default.Table_d07400a150ac4a6c86bac1caeebe8739 [id=fe2df9a958ec4863ab2100750ba7dc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.451710 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe922c438a894a7aa04699b2d2897a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.452394 23503 tablet_service.cc:1505] Processing CreateTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf (DEFAULT_TABLE table=default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.452942 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5964da70989b4e6cb77fe5d0a52ce9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.454836 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf (DEFAULT_TABLE table=default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.456377 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5964da70989b4e6cb77fe5d0a52ce9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.459172 23787 tablet_bootstrap.cc:492] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.459342 23787 ts_tablet_manager.cc:1403] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.001s
I20251023 20:15:17.460115 23787 raft_consensus.cc:359] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.460387 23787 raft_consensus.cc:385] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.460479 23787 raft_consensus.cc:740] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.460711 23787 consensus_queue.cc:260] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.464128 23787 ts_tablet_manager.cc:1434] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:17.464440 23787 tablet_bootstrap.cc:492] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.465605 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.469976 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf (DEFAULT_TABLE table=default.Table_1fd972f3964245fab674e2cfc26a8210 [id=cf5e426caddf4508bc0780b9809fff5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.470202 23786 tablet_bootstrap.cc:492] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:17.470420 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.470633 23786 ts_tablet_manager.cc:1403] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.008s
I20251023 20:15:17.471966 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 (DEFAULT_TABLE table=default.Table_c27155507a7f45cc860ea6089419e20e [id=4bbc0f95fee64c7c8570961ba3f1c0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.472685 23786 raft_consensus.cc:359] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.472959 23786 raft_consensus.cc:385] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.473042 23786 raft_consensus.cc:740] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.473380 23786 consensus_queue.cc:260] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.473874 23786 ts_tablet_manager.cc:1434] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:17.474362 23786 tablet_bootstrap.cc:492] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:17.475303 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 0438593fbbdf47cebfb64121acad104a
I20251023 20:15:17.475642 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 6e1b4748c0244b258960b1025da47562
I20251023 20:15:17.476846 23786 tablet_bootstrap.cc:654] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.477329 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 8289ce37fa074891beb584bf2fff9747
I20251023 20:15:17.477767 23143 catalog_manager.cc:6719] ReplaceTablet: tablet 8289ce37fa074891beb584bf2fff9747 deleted and replaced by tablet c24c33c11d0d4d22bdcb96e1cf88ad66
I20251023 20:15:17.479836 23786 tablet_bootstrap.cc:492] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:17.479981 23786 ts_tablet_manager.cc:1403] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20251023 20:15:17.480700 23786 raft_consensus.cc:359] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.480870 23786 raft_consensus.cc:385] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.480954 23786 raft_consensus.cc:740] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.481192 23786 consensus_queue.cc:260] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.481314 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fa6a838b1bd409ba2a4e2074576ee23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.481647 23786 ts_tablet_manager.cc:1434] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:17.481904 23786 tablet_bootstrap.cc:492] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:17.482831 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (replaced by tablet c24c33c11d0d4d22bdcb96e1cf88ad66) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.482895 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.483058 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.483420 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
I20251023 20:15:17.483639 23786 tablet_bootstrap.cc:654] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:17.485112 23928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:17.485556 23801 raft_consensus.cc:493] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:17.486780 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e0f8d8184a68453abd2792ccc7740aa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f8d8184a68453abd2792ccc7740aa1 already in progress: creating tablet
I20251023 20:15:17.486919 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b3b5199d1faf4e5795ba391229812ea9" } modify_external_catalogs: true
I20251023 20:15:17.487088 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b3b5199d1faf4e5795ba391229812ea9" } modify_external_catalogs: true
I20251023 20:15:17.487274 23784 tablet_replica.cc:333] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:17.487524 23784 raft_consensus.cc:2243] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.487754 23784 raft_consensus.cc:2272] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.487985 23370 tablet_service.cc:1505] Processing CreateTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c (DEFAULT_TABLE table=default.Table_4e18a968658f470d870bc932102ebf00 [id=ad634b38921149658d6212b02c421514]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.488857 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf (DEFAULT_TABLE table=default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.488889 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 (DEFAULT_TABLE table=default.Table_e926815d67914fed915b184838f05b72 [id=265a9707ddea4ee1a19aac6033883822]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.489114 23379 tablet_service.cc:1505] Processing CreateTablet for tablet fe922c438a894a7aa04699b2d2897a28 (DEFAULT_TABLE table=default.Table_d07400a150ac4a6c86bac1caeebe8739 [id=fe2df9a958ec4863ab2100750ba7dc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.489197 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.489997 23784 ts_tablet_manager.cc:1916] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.487866 23801 raft_consensus.cc:515] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.491354 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5964da70989b4e6cb77fe5d0a52ce9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.491990 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (replaced by tablet c24c33c11d0d4d22bdcb96e1cf88ad66) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.492187 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.492319 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2273d5581784756b737f8c7c17ed3a9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:17.492571 23536 raft_consensus.cc:2468] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:17.489732 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 5346a221c78342359eb6e418bd12cd24 (DEFAULT_TABLE table=default.Table_914e9b5e4a7443c5baf677b4b6857359 [id=71ba16e8b75642e09686eb46c4315366]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.494081 23827 server_base.cc:1047] running on GCE node
W20251023 20:15:17.494498 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 0438593fbbdf47cebfb64121acad104a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0438593fbbdf47cebfb64121acad104a already in progress: creating tablet
I20251023 20:15:17.495160 23827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:17.497272 23786 tablet_bootstrap.cc:492] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:17.497427 23786 ts_tablet_manager.cc:1403] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20251023 20:15:17.497491 23143 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:17.497843 23146 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998]). Expected version 4 got 0
W20251023 20:15:17.498057 23143 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 34a738510f514b088469de3d8a39ca2c consensus state has no leader
I20251023 20:15:17.498200 23786 raft_consensus.cc:359] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.498381 23786 raft_consensus.cc:385] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.498440 23786 raft_consensus.cc:740] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.498668 23786 consensus_queue.cc:260] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.499112 23786 ts_tablet_manager.cc:1434] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:17.499433 23786 tablet_bootstrap.cc:492] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:17.502956 23503 tablet_service.cc:1505] Processing CreateTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 (DEFAULT_TABLE table=default.Table_d2b87a7247624b0380ee21ec002b9f77 [id=f64a3298a8d44a7ba1700cac9bf91080]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.505785 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe922c438a894a7aa04699b2d2897a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.506635 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.506909 23801 leader_election.cc:290] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:17.507588 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7a4278caa64d4373b42c8b865ea6c2d8" } modify_external_catalogs: true
I20251023 20:15:17.507771 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.507835 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7a4278caa64d4373b42c8b865ea6c2d8" } modify_external_catalogs: true
I20251023 20:15:17.507951 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:17.508275 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.509215 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.509680 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.510219 23637 tablet_service.cc:1505] Processing CreateTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 (DEFAULT_TABLE table=default.Table_d2b87a7247624b0380ee21ec002b9f77 [id=f64a3298a8d44a7ba1700cac9bf91080]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.510665 23604 leader_election.cc:304] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:17.511085 23801 raft_consensus.cc:2804] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:17.511796 23363 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (replaced by tablet c24c33c11d0d4d22bdcb96e1cf88ad66) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.512149 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet 955412de8bd1482ea7304872b48f38c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 283166bbd8004ce3bca9f2ecbdbcbf48) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.512180 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2273d5581784756b737f8c7c17ed3a9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:17.505486 23786 tablet_bootstrap.cc:654] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:17.512980 23132 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9d1ddde7e3134e23b3a9356523a93753 (table default.Table_a1332462902c4ea5971afeb4c3688865 [id=f27defb723394b4db220b278b92e6ae8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9d1ddde7e3134e23b3a9356523a93753 consensus state has no leader
I20251023 20:15:17.513317 23785 tablet_replica.cc:333] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:17.513619 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.514884 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet cc5ce157b6ca42da90b051a184983bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.516016 23500 tablet_service.cc:1505] Processing CreateTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb (DEFAULT_TABLE table=default.Table_b3b5199d1faf4e5795ba391229812ea9 [id=8ec2443a50674a67a3393236ee4bc48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.516289 23139 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c24c33c11d0d4d22bdcb96e1cf88ad66
I20251023 20:15:17.516376 23500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b9f4c5d9e0469c87071a33722ce3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.517366 23827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:17.518950 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 86b9f4c5d9e0469c87071a33722ce3eb
I20251023 20:15:17.519642 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.520608 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 3a7306245d7b41cfab795450761144f5 (DEFAULT_TABLE table=default.Table_59bd96856079400e8eda1b518a1ec457 [id=adeeb85b87d747bd836be0b7c403ed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.520972 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7306245d7b41cfab795450761144f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.521791 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 37c8403b53f04a8e97ffe058e59b425c (DEFAULT_TABLE table=default.Table_191b94017cca4cfb946f38ab19ebfe20 [id=330c829b79fd4cce881fc565a3463863]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.522106 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c8403b53f04a8e97ffe058e59b425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.522811 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet ae388da9e4f94ba5af82a67088ab7097
W20251023 20:15:17.524077 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 6e1b4748c0244b258960b1025da47562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e1b4748c0244b258960b1025da47562 already in progress: creating tablet
I20251023 20:15:17.525429 23369 tablet_service.cc:1505] Processing CreateTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 (DEFAULT_TABLE table=default.Table_d2b87a7247624b0380ee21ec002b9f77 [id=f64a3298a8d44a7ba1700cac9bf91080]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.525843 23364 tablet_service.cc:1505] Processing CreateTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 (DEFAULT_TABLE table=default.Table_1fc3a0e0fe484168b2f767fdcc3eba64 [id=83f2c96018ba47e79c3c7ad4df2852a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.513631 23801 raft_consensus.cc:493] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:17.526206 23362 tablet_service.cc:1505] Processing CreateTablet for tablet d82c92e863b2470699cc822b8cb8c14d (DEFAULT_TABLE table=default.Table_8ac87d9615a74c5c858be1d6c31772a6 [id=1740292e42a544e5bd5d834a281bddbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.526536 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ba7ab28dfa4ac38e0b58ba2115495c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.505832 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 (DEFAULT_TABLE table=default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.509308 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.510445 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 5416da30c66645c19d36112980608527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.511142 23510 tablet_service.cc:1505] Processing CreateTablet for tablet d82c92e863b2470699cc822b8cb8c14d (DEFAULT_TABLE table=default.Table_8ac87d9615a74c5c858be1d6c31772a6 [id=1740292e42a544e5bd5d834a281bddbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.529470 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 925e53d9ea6f4c7baa38551eb917828d (DEFAULT_TABLE table=default.Table_da2bae6bea6047318cd9fb653b35475c [id=275f37e241644f14bba90f90529ebdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.529702 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a708cd011802458aa0e2991fd4b4a2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.530000 23827 hybrid_clock.cc:648] HybridClock initialized: now 1761250517529915 us; error 52 us; skew 500 ppm
I20251023 20:15:17.530503 23827 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 20:15:17.531630 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet d8ad21e3888a4915ae5f252c2fca64cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.531782 23373 tablet_service.cc:1505] Processing CreateTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 (DEFAULT_TABLE table=default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.532212 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet dc8f615c621043da88d52f3964b0a0ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.532292 23363 tablet_service.cc:1552] Processing DeleteTablet for tablet bd88285bf0a94fad858a25004609131c with delete_type TABLET_DATA_DELETED (replaced by tablet c317dc31cb0448d985512e6ba98735be) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.532559 23790 tablet_bootstrap.cc:492] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.532704 23790 ts_tablet_manager.cc:1403] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.122s user 0.003s sys 0.019s
I20251023 20:15:17.533216 23785 raft_consensus.cc:2243] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.533615 23801 raft_consensus.cc:3060] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:17.511602 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 (DEFAULT_TABLE table=default.Table_1fc3a0e0fe484168b2f767fdcc3eba64 [id=83f2c96018ba47e79c3c7ad4df2852a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.534106 23783 ts_tablet_manager.cc:1929] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:17.534271 23605 leader_election.cc:343] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:17.534490 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 5e73ae95c73945f081c298f49d47c904 (DEFAULT_TABLE table=default.Table_8d8ada20a7bb43eb9a534aedca23cfd2 [id=6fb3e731496a466a918ab84247724a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.534935 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82c92e863b2470699cc822b8cb8c14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.536906 23785 raft_consensus.cc:2272] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.537230 23787 tablet_bootstrap.cc:654] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.538231 23783 log.cc:1199] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/5416da30c66645c19d36112980608527
I20251023 20:15:17.538283 23639 tablet_service.cc:1505] Processing CreateTablet for tablet d82c92e863b2470699cc822b8cb8c14d (DEFAULT_TABLE table=default.Table_8ac87d9615a74c5c858be1d6c31772a6 [id=1740292e42a544e5bd5d834a281bddbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.539968 23790 raft_consensus.cc:359] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.540441 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb (DEFAULT_TABLE table=default.Table_b3b5199d1faf4e5795ba391229812ea9 [id=8ec2443a50674a67a3393236ee4bc48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.541862 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5346a221c78342359eb6e418bd12cd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.541882 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40e0301fb5974ec5b6e8291230a736c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.543329 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4f168f47af364b3db52fbbc64de90a6e" } new_table_name: "default.Table_d38870cf316141849bc0120ccafa0dba" modify_external_catalogs: true
I20251023 20:15:17.543448 23790 raft_consensus.cc:385] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.543490 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 925e53d9ea6f4c7baa38551eb917828d (DEFAULT_TABLE table=default.Table_da2bae6bea6047318cd9fb653b35475c [id=275f37e241644f14bba90f90529ebdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.545536 23790 raft_consensus.cc:740] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.545801 23790 consensus_queue.cc:260] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.548348 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e73ae95c73945f081c298f49d47c904. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.548883 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:17.549788 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c317dc31cb0448d985512e6ba98735be consensus state has no leader
I20251023 20:15:17.550400 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925e53d9ea6f4c7baa38551eb917828d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.552124 23827 webserver.cc:492] Webserver started at http://127.22.127.190:44199/ using document root <none> and password file <none>
I20251023 20:15:17.552614 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40e0301fb5974ec5b6e8291230a736c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.553181 23785 ts_tablet_manager.cc:1916] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.553269 23827 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:17.553402 23827 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:17.553753 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 3a7306245d7b41cfab795450761144f5 (DEFAULT_TABLE table=default.Table_59bd96856079400e8eda1b518a1ec457 [id=adeeb85b87d747bd836be0b7c403ed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.554175 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b9f4c5d9e0469c87071a33722ce3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.556173 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82c92e863b2470699cc822b8cb8c14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.538625 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 (DEFAULT_TABLE table=default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.559401 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_bdb5b8891a814c998413f5b7a0a605cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.559605 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 (DEFAULT_TABLE table=default.Table_1fc3a0e0fe484168b2f767fdcc3eba64 [id=83f2c96018ba47e79c3c7ad4df2852a7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:17.559978 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdb5b8891a814c998413f5b7a0a605cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.559744 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 951b2276d3324c2d999c8d7fb97767bc (DEFAULT_TABLE table=default.Table_441bf8050be9453c85aef2cb7ac63ea4 [id=425e2178647a42aaa99cef01f06523f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.562609 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d38870cf316141849bc0120ccafa0dba" } new_table_name: "default.Table_ff93d93fcfce4bd686e1e6fa95a31ecc" modify_external_catalogs: true
I20251023 20:15:17.563071 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b6c3e7169154e3497dd3570dca0d745. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.563086 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c27155507a7f45cc860ea6089419e20e" } new_table_name: "default.Table_05a9ffe8a73a4527be8ac2ba2fecc052" modify_external_catalogs: true
I20251023 20:15:17.565239 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_07cfd6b64da94bbfa85cb6c533f550c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.565757 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07cfd6b64da94bbfa85cb6c533f550c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.565924 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 951b2276d3324c2d999c8d7fb97767bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.565979 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82c92e863b2470699cc822b8cb8c14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.566849 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b380171aa08d439eadeddb7b4c6e6a6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.567286 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b380171aa08d439eadeddb7b4c6e6a6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.570685 23364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b6c3e7169154e3497dd3570dca0d745. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.571919 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.572474 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40e0301fb5974ec5b6e8291230a736c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.574177 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb (DEFAULT_TABLE table=default.Table_b3b5199d1faf4e5795ba391229812ea9 [id=8ec2443a50674a67a3393236ee4bc48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.575227 23783 ts_tablet_manager.cc:1950] T 5416da30c66645c19d36112980608527 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.576119 23783 tablet_replica.cc:333] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.576515 23783 raft_consensus.cc:2243] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.576689 23783 raft_consensus.cc:2272] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.577294 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 37c8403b53f04a8e97ffe058e59b425c (DEFAULT_TABLE table=default.Table_191b94017cca4cfb946f38ab19ebfe20 [id=330c829b79fd4cce881fc565a3463863]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.577490 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b6c3e7169154e3497dd3570dca0d745. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.577550 23801 raft_consensus.cc:515] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.572525 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 925e53d9ea6f4c7baa38551eb917828d (DEFAULT_TABLE table=default.Table_da2bae6bea6047318cd9fb653b35475c [id=275f37e241644f14bba90f90529ebdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.578290 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925e53d9ea6f4c7baa38551eb917828d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.578599 23801 leader_election.cc:290] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:17.578859 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b9f4c5d9e0469c87071a33722ce3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.579102 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2273d5581784756b737f8c7c17ed3a9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:17.581472 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a708cd011802458aa0e2991fd4b4a2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.582029 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b7a18af7518d43c6b31ac4abed6372c2" } new_table_name: "default.Table_3a387ab7d11047b392362bbe1fa7f701" modify_external_catalogs: true
W20251023 20:15:17.583212 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c317dc31cb0448d985512e6ba98735be consensus state has no leader
I20251023 20:15:17.584718 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 3a7306245d7b41cfab795450761144f5 (DEFAULT_TABLE table=default.Table_59bd96856079400e8eda1b518a1ec457 [id=adeeb85b87d747bd836be0b7c403ed13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.585243 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 951b2276d3324c2d999c8d7fb97767bc (DEFAULT_TABLE table=default.Table_441bf8050be9453c85aef2cb7ac63ea4 [id=425e2178647a42aaa99cef01f06523f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.585870 23536 raft_consensus.cc:3060] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:17.586252 23790 ts_tablet_manager.cc:1434] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.037s user 0.002s sys 0.001s
I20251023 20:15:17.588406 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 37c8403b53f04a8e97ffe058e59b425c (DEFAULT_TABLE table=default.Table_191b94017cca4cfb946f38ab19ebfe20 [id=330c829b79fd4cce881fc565a3463863]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.588799 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c8403b53f04a8e97ffe058e59b425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.588950 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.592294 23783 ts_tablet_manager.cc:1916] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.592448 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 5e73ae95c73945f081c298f49d47c904 (DEFAULT_TABLE table=default.Table_8d8ada20a7bb43eb9a534aedca23cfd2 [id=6fb3e731496a466a918ab84247724a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.597332 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8370048ef7d8400c9ea8f72c19305e61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.597791 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3fd19e6c8ca54391bd33c1c235d72938" } new_table_name: "default.Table_5abd17a4558f4f81a956cdf6df0d4524" modify_external_catalogs: true
W20251023 20:15:17.597913 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8370048ef7d8400c9ea8f72c19305e61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.598527 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e62c8689605f4db7b6e5016d5dcda175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.599012 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e62c8689605f4db7b6e5016d5dcda175 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.599900 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_578c34f1b0124b0296648c859e5904cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.600328 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_578c34f1b0124b0296648c859e5904cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.601862 23501 tablet_service.cc:1505] Processing CreateTablet for tablet cb915d114fc0421999d079c23cad8eb5 (DEFAULT_TABLE table=default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.602375 23642 tablet_service.cc:1505] Processing CreateTablet for tablet cb915d114fc0421999d079c23cad8eb5 (DEFAULT_TABLE table=default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.602896 23372 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.602948 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 5416da30c66645c19d36112980608527 (table default.Table_b6760837eeee4b5082522107b87d72f5 [id=6e38fec2eebd48b8a6364be16ec99997]) successfully deleted
I20251023 20:15:17.603963 23786 tablet_bootstrap.cc:492] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:17.604327 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e73ae95c73945f081c298f49d47c904. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:17.605475 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fa6a838b1bd409ba2a4e2074576ee23 consensus state has no leader
W20251023 20:15:17.606787 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 215e21ca449c481d8716783a65105101 consensus state has no leader
I20251023 20:15:17.608286 23790 tablet_bootstrap.cc:492] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.610993 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7306245d7b41cfab795450761144f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.612331 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925e53d9ea6f4c7baa38551eb917828d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.612666 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c8403b53f04a8e97ffe058e59b425c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.612352 23375 tablet_service.cc:1505] Processing CreateTablet for tablet cb915d114fc0421999d079c23cad8eb5 (DEFAULT_TABLE table=default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.614253 23794 raft_consensus.cc:493] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.614439 23794 raft_consensus.cc:515] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.615116 23794 leader_election.cc:290] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:17.615638 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 5e73ae95c73945f081c298f49d47c904 (DEFAULT_TABLE table=default.Table_8d8ada20a7bb43eb9a534aedca23cfd2 [id=6fb3e731496a466a918ab84247724a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.616000 23786 ts_tablet_manager.cc:1403] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.106s user 0.010s sys 0.011s
W20251023 20:15:17.617136 23470 leader_election.cc:343] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 5416da30c66645c19d36112980608527
I20251023 20:15:17.618204 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 951b2276d3324c2d999c8d7fb97767bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.618651 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a708cd011802458aa0e2991fd4b4a2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.618724 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb915d114fc0421999d079c23cad8eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.619252 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2273d5581784756b737f8c7c17ed3a9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:17.619460 23399 raft_consensus.cc:3060] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:17.619810 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ee55921ad9e8422984b6afd71da0f4ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.620358 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee55921ad9e8422984b6afd71da0f4ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.623447 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb915d114fc0421999d079c23cad8eb5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:17.630023 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4778436aed64b53adb2bab54bcf263b consensus state has no leader
I20251023 20:15:17.630491 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e73ae95c73945f081c298f49d47c904. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.631815 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_13473968f0134166b1760d69c86b8b2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.632405 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13473968f0134166b1760d69c86b8b2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.641042 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 951b2276d3324c2d999c8d7fb97767bc (DEFAULT_TABLE table=default.Table_441bf8050be9453c85aef2cb7ac63ea4 [id=425e2178647a42aaa99cef01f06523f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.641510 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.641731 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb915d114fc0421999d079c23cad8eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.645310 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 951b2276d3324c2d999c8d7fb97767bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.646025 23536 raft_consensus.cc:2468] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:17.646649 23604 leader_election.cc:304] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:17.647573 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a7306245d7b41cfab795450761144f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.647483 23801 raft_consensus.cc:2804] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:17.656255 23783 ts_tablet_manager.cc:1929] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.656482 23783 log.cc:1199] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/50c75cce4b9d4c48800b2d0908676a14
I20251023 20:15:17.657395 23783 ts_tablet_manager.cc:1950] T 50c75cce4b9d4c48800b2d0908676a14 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.658432 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 50c75cce4b9d4c48800b2d0908676a14 (table default.Table_39878ca4314e43b8bf37b4cba1a5fb54 [id=c1b3327a41c8461cb83d9b31afa03e4d]) successfully deleted
I20251023 20:15:17.663888 23787 tablet_bootstrap.cc:492] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.665038 23787 ts_tablet_manager.cc:1403] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.200s user 0.007s sys 0.003s
I20251023 20:15:17.673914 23787 raft_consensus.cc:359] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.674147 23787 raft_consensus.cc:385] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.674245 23787 raft_consensus.cc:740] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.674760 23787 consensus_queue.cc:260] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.636499 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5416da30c66645c19d36112980608527" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:17.678377 23471 leader_election.cc:343] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:17.678611 23471 leader_election.cc:304] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:17.678898 23794 raft_consensus.cc:2749] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:17.681286 23783 tablet_replica.cc:333] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:17.681838 23783 raft_consensus.cc:2243] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.682868 23783 raft_consensus.cc:2272] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.699074 23810 raft_consensus.cc:493] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.699239 23810 raft_consensus.cc:515] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.704181 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 (DEFAULT_TABLE table=default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0 [id=643c30acd68743bfb7c0ebd7b98199e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.704994 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a64a7c2e1f242e1b54243489622bae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.705298 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 252b5480222f49439052c86a4115cc9b (DEFAULT_TABLE table=default.Table_bdb5b8891a814c998413f5b7a0a605cc [id=0674b989e3f74438b47e1fc653f5db49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.704620 23637 tablet_service.cc:1505] Processing CreateTablet for tablet d6b0a4db18df490a966e8c475ddd14f4 (DEFAULT_TABLE table=default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12 [id=6eda60ed5ff344f3a610b8139aa5a6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.705667 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252b5480222f49439052c86a4115cc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.705900 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b0a4db18df490a966e8c475ddd14f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.706290 23503 tablet_service.cc:1505] Processing CreateTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 (DEFAULT_TABLE table=default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0 [id=643c30acd68743bfb7c0ebd7b98199e7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:17.708289 23333 leader_election.cc:343] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: dc8f615c621043da88d52f3964b0a0ec
W20251023 20:15:17.708562 23333 leader_election.cc:343] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: dc8f615c621043da88d52f3964b0a0ec
I20251023 20:15:17.708722 23333 leader_election.cc:304] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:17.713333 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a64a7c2e1f242e1b54243489622bae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.716243 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_5abd17a4558f4f81a956cdf6df0d4524" } modify_external_catalogs: true
I20251023 20:15:17.716447 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_5abd17a4558f4f81a956cdf6df0d4524" } modify_external_catalogs: true
I20251023 20:15:17.716521 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e186a061f84042cd82645b37beae1da1" } modify_external_catalogs: true
I20251023 20:15:17.716703 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e186a061f84042cd82645b37beae1da1" } modify_external_catalogs: true
I20251023 20:15:17.718951 23827 fs_manager.cc:714] Time spent opening directory manager: real 0.163s user 0.029s sys 0.027s
I20251023 20:15:17.721223 23783 ts_tablet_manager.cc:1916] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.725348 23810 leader_election.cc:290] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:17.725404 23801 raft_consensus.cc:697] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:17.725561 23810 raft_consensus.cc:2749] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:17.726369 23801 consensus_queue.cc:237] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.729259 23787 ts_tablet_manager.cc:1434] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.056s user 0.005s sys 0.001s
I20251023 20:15:17.729784 23787 tablet_bootstrap.cc:492] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.732056 23787 tablet_bootstrap.cc:654] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.732234 23644 tablet_service.cc:1505] Processing CreateTablet for tablet f2a27706ecdb4ab78bb96c10a9f6757f (DEFAULT_TABLE table=default.Table_13473968f0134166b1760d69c86b8b2f [id=8a388a26b69d416ab99d15c024f1d3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.733116 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a27706ecdb4ab78bb96c10a9f6757f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.735488 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 23225d31c2e647398aa58494c627507e from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.736693 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bdb5b8891a814c998413f5b7a0a605cc" } modify_external_catalogs: true
I20251023 20:15:17.736899 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bdb5b8891a814c998413f5b7a0a605cc" } modify_external_catalogs: true
I20251023 20:15:17.736897 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 0858358aabff47a69dc77f4669b225c5 (DEFAULT_TABLE table=default.Table_07cfd6b64da94bbfa85cb6c533f550c5 [id=c36b140eefe1435d93b7ba48ca07e4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.737213 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e8cec02c09a44acab531a2f71ebb0a73" } modify_external_catalogs: true
I20251023 20:15:17.737342 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0858358aabff47a69dc77f4669b225c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.737407 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e8cec02c09a44acab531a2f71ebb0a73" } modify_external_catalogs: true
I20251023 20:15:17.739565 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4dbe3b1770374c18821af919d529fc09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.740093 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dbe3b1770374c18821af919d529fc09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.736907 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet 66a2f3adb5d347a0adc198cd1ef93304 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.740711 23647 tablet_service.cc:1505] Processing CreateTablet for tablet c97e2e0f839a4ef389fe100b6a81301e (DEFAULT_TABLE table=default.Table_578c34f1b0124b0296648c859e5904cb [id=55a8a0300c454094a6ee9c2c7897495c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.741122 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97e2e0f839a4ef389fe100b6a81301e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.741775 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 15ced74d0dd74e5195ed98358957d0dd (DEFAULT_TABLE table=default.Table_e62c8689605f4db7b6e5016d5dcda175 [id=91e8d69e8f264c6dafd1173e05f7be06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.742205 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15ced74d0dd74e5195ed98358957d0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.742527 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d2b87a7247624b0380ee21ec002b9f77" } modify_external_catalogs: true
I20251023 20:15:17.742826 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d2b87a7247624b0380ee21ec002b9f77" } modify_external_catalogs: true
I20251023 20:15:17.742967 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c01f9e9030974e2e8b43da4e68ef0145"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.743491 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c01f9e9030974e2e8b43da4e68ef0145 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.743480 23786 raft_consensus.cc:359] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.743709 23786 raft_consensus.cc:385] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.745285 23641 tablet_service.cc:1505] Processing CreateTablet for tablet d951134f6773410386bc57f7f90310c3 (DEFAULT_TABLE table=default.Table_8370048ef7d8400c9ea8f72c19305e61 [id=7313119e56d94c05b9d8a31709af6ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.745795 23399 raft_consensus.cc:2468] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:17.746347 23786 raft_consensus.cc:740] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.746632 23786 consensus_queue.cc:260] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.749274 23790 tablet_bootstrap.cc:654] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.750571 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 (DEFAULT_TABLE table=default.Table_ee55921ad9e8422984b6afd71da0f4ee [id=c6c3d95bdfef4028a356f1365e9fa53d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.750965 23514 tablet_service.cc:1505] Processing CreateTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc (DEFAULT_TABLE table=default.Table_b380171aa08d439eadeddb7b4c6e6a6a [id=5746055cb51b4ee4b54b672987471cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.752511 23504 tablet_service.cc:1505] Processing CreateTablet for tablet d6b0a4db18df490a966e8c475ddd14f4 (DEFAULT_TABLE table=default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12 [id=6eda60ed5ff344f3a610b8139aa5a6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.756332 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet f4778436aed64b53adb2bab54bcf263b
I20251023 20:15:17.757777 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 252b5480222f49439052c86a4115cc9b
I20251023 20:15:17.760450 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 0858358aabff47a69dc77f4669b225c5 (DEFAULT_TABLE table=default.Table_07cfd6b64da94bbfa85cb6c533f550c5 [id=c36b140eefe1435d93b7ba48ca07e4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.762759 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e08dad1d0f643efb3a821a1b9196290. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.763005 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c5b69df4c04365b1ac1457fe8300cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.765822 23785 ts_tablet_manager.cc:1929] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.766796 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a2dbb7997d0f4e6bac48cfb196f5ccb6
I20251023 20:15:17.769378 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:17.769843 23144 catalog_manager.cc:6719] ReplaceTablet: tablet 66a2f3adb5d347a0adc198cd1ef93304 deleted and replaced by tablet b680b1cbc103403b91a1104248ecbdcf
I20251023 20:15:17.770470 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 4ea63540149c4c02abb8ae51209060a3
I20251023 20:15:17.773561 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.774662 23786 ts_tablet_manager.cc:1434] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.138s user 0.002s sys 0.000s
I20251023 20:15:17.774765 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.774425 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 15ced74d0dd74e5195ed98358957d0dd (DEFAULT_TABLE table=default.Table_e62c8689605f4db7b6e5016d5dcda175 [id=91e8d69e8f264c6dafd1173e05f7be06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.775507 23786 tablet_bootstrap.cc:492] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:17.774896 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 with delete_type TABLET_DATA_DELETED (replaced by tablet e5368813164949ff80e27bf59dcc931a) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.775511 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.777928 23500 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.779639 23785 log.cc:1199] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/955412de8bd1482ea7304872b48f38c0
I20251023 20:15:17.780099 23785 ts_tablet_manager.cc:1950] T 955412de8bd1482ea7304872b48f38c0 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:17.781328 23785 tablet_replica.cc:333] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:17.781463 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 252b5480222f49439052c86a4115cc9b (DEFAULT_TABLE table=default.Table_bdb5b8891a814c998413f5b7a0a605cc [id=0674b989e3f74438b47e1fc653f5db49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.782829 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a708cd011802458aa0e2991fd4b4a2d3
I20251023 20:15:17.783308 23785 raft_consensus.cc:2243] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.783725 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0858358aabff47a69dc77f4669b225c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.784703 23785 raft_consensus.cc:2272] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:17.781603 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 73047.2us
I20251023 20:15:17.785917 23505 tablet_service.cc:1505] Processing CreateTablet for tablet d951134f6773410386bc57f7f90310c3 (DEFAULT_TABLE table=default.Table_8370048ef7d8400c9ea8f72c19305e61 [id=7313119e56d94c05b9d8a31709af6ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.786970 23368 tablet_service.cc:1505] Processing CreateTablet for tablet d6b0a4db18df490a966e8c475ddd14f4 (DEFAULT_TABLE table=default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12 [id=6eda60ed5ff344f3a610b8139aa5a6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.787405 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b0a4db18df490a966e8c475ddd14f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.787518 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15ced74d0dd74e5195ed98358957d0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.787521 23377 tablet_service.cc:1505] Processing CreateTablet for tablet f2a27706ecdb4ab78bb96c10a9f6757f (DEFAULT_TABLE table=default.Table_13473968f0134166b1760d69c86b8b2f [id=8a388a26b69d416ab99d15c024f1d3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.787839 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a27706ecdb4ab78bb96c10a9f6757f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.786965 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 252b5480222f49439052c86a4115cc9b (DEFAULT_TABLE table=default.Table_bdb5b8891a814c998413f5b7a0a605cc [id=0674b989e3f74438b47e1fc653f5db49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.789516 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252b5480222f49439052c86a4115cc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.790127 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.792928 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252b5480222f49439052c86a4115cc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.793294 23508 tablet_service.cc:1505] Processing CreateTablet for tablet f2a27706ecdb4ab78bb96c10a9f6757f (DEFAULT_TABLE table=default.Table_13473968f0134166b1760d69c86b8b2f [id=8a388a26b69d416ab99d15c024f1d3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.793721 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 (DEFAULT_TABLE table=default.Table_ee55921ad9e8422984b6afd71da0f4ee [id=c6c3d95bdfef4028a356f1365e9fa53d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.794086 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e08dad1d0f643efb3a821a1b9196290. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.794857 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d951134f6773410386bc57f7f90310c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.796818 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d951134f6773410386bc57f7f90310c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.798067 23786 tablet_bootstrap.cc:654] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.798106 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2a27706ecdb4ab78bb96c10a9f6757f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.799575 23369 tablet_service.cc:1505] Processing CreateTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc (DEFAULT_TABLE table=default.Table_b380171aa08d439eadeddb7b4c6e6a6a [id=5746055cb51b4ee4b54b672987471cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.799961 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c5b69df4c04365b1ac1457fe8300cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.800365 23784 ts_tablet_manager.cc:1929] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.800493 23784 log.cc:1199] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/949e31d4a9ee4579843ed54153e22c7d
I20251023 20:15:17.800820 23784 ts_tablet_manager.cc:1950] T 949e31d4a9ee4579843ed54153e22c7d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:17.804548 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b0a4db18df490a966e8c475ddd14f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.808766 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 0858358aabff47a69dc77f4669b225c5 (DEFAULT_TABLE table=default.Table_07cfd6b64da94bbfa85cb6c533f550c5 [id=c36b140eefe1435d93b7ba48ca07e4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.808926 23637 tablet_service.cc:1505] Processing CreateTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc (DEFAULT_TABLE table=default.Table_b380171aa08d439eadeddb7b4c6e6a6a [id=5746055cb51b4ee4b54b672987471cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.810709 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 949e31d4a9ee4579843ed54153e22c7d (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]) successfully deleted
I20251023 20:15:17.812798 23500 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:17.813898 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 0438593fbbdf47cebfb64121acad104a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0438593fbbdf47cebfb64121acad104a already in progress: creating tablet
W20251023 20:15:17.814147 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6e1b4748c0244b258960b1025da47562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e1b4748c0244b258960b1025da47562 already in progress: creating tablet
I20251023 20:15:17.814249 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 15ced74d0dd74e5195ed98358957d0dd (DEFAULT_TABLE table=default.Table_e62c8689605f4db7b6e5016d5dcda175 [id=91e8d69e8f264c6dafd1173e05f7be06]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.815428 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15ced74d0dd74e5195ed98358957d0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.816462 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c5b69df4c04365b1ac1457fe8300cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.817167 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.817876 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.818034 23501 tablet_service.cc:1505] Processing CreateTablet for tablet c97e2e0f839a4ef389fe100b6a81301e (DEFAULT_TABLE table=default.Table_578c34f1b0124b0296648c859e5904cb [id=55a8a0300c454094a6ee9c2c7897495c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.818413 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97e2e0f839a4ef389fe100b6a81301e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.822240 23785 ts_tablet_manager.cc:1916] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.822909 23143 catalog_manager.cc:5649] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:17.824194 23143 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:17.826440 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0858358aabff47a69dc77f4669b225c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.828011 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 (DEFAULT_TABLE table=default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0 [id=643c30acd68743bfb7c0ebd7b98199e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.835963 23783 ts_tablet_manager.cc:1929] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.837659 23787 tablet_bootstrap.cc:492] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:17.837988 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet f4778436aed64b53adb2bab54bcf263b from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.838131 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 6558265ff9e249c68123577958f1401a from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.838500 23784 tablet_replica.cc:333] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:17.839385 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bb6092ff59da4b9988c720c4987d7c35" } new_table_name: "default.Table_a21f26a0638f42d682abb0448b2a53e5" modify_external_catalogs: true
I20251023 20:15:17.839923 23947 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:17.840163 23784 raft_consensus.cc:2243] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.840369 23784 raft_consensus.cc:2272] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.840891 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_13473968f0134166b1760d69c86b8b2f" } new_table_name: "default.Table_a216eab77a9f465982dad2f085170ddf" modify_external_catalogs: true
I20251023 20:15:17.842324 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f5ea72074b174da28b3738378167f49c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.842860 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5ea72074b174da28b3738378167f49c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.843243 23363 tablet_service.cc:1505] Processing CreateTablet for tablet d951134f6773410386bc57f7f90310c3 (DEFAULT_TABLE table=default.Table_8370048ef7d8400c9ea8f72c19305e61 [id=7313119e56d94c05b9d8a31709af6ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.843984 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7c03bf4f1b034b139999af6a349dccf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.844414 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c03bf4f1b034b139999af6a349dccf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.844897 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8310cc42e75240cf843814583db61948" } new_table_name: "default.Table_b4dc4f9658ce4837999abcf5720cea58" modify_external_catalogs: true
I20251023 20:15:17.845322 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a1c9b5357e504ba2a116c58ff86ad13c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.845583 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a64a7c2e1f242e1b54243489622bae5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:17.845827 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1c9b5357e504ba2a116c58ff86ad13c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.846269 23784 ts_tablet_manager.cc:1916] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.846719 23827 fs_manager.cc:730] Time spent opening block manager: real 0.125s user 0.009s sys 0.016s
I20251023 20:15:17.847031 23374 tablet_service.cc:1505] Processing CreateTablet for tablet c97e2e0f839a4ef389fe100b6a81301e (DEFAULT_TABLE table=default.Table_578c34f1b0124b0296648c859e5904cb [id=55a8a0300c454094a6ee9c2c7897495c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.847843 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.848084 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.848209 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d951134f6773410386bc57f7f90310c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.848321 23783 log.cc:1199] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/73b4abd308054bac9d49b1d8e7dd2ebc
I20251023 20:15:17.850447 23143 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:17.851038 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.846993 23827 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
uuid: "6545140d94b7485aa81c51ad233355cd"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
I20251023 20:15:17.851780 23827 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:17.852435 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c97e2e0f839a4ef389fe100b6a81301e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.855350 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.855852 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.856004 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.856709 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.858273 23787 ts_tablet_manager.cc:1403] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.008s
W20251023 20:15:17.863358 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 86b9f4c5d9e0469c87071a33722ce3eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86b9f4c5d9e0469c87071a33722ce3eb already in progress: creating tablet
I20251023 20:15:17.864182 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 6558265ff9e249c68123577958f1401a
I20251023 20:15:17.864694 23140 catalog_manager.cc:6719] ReplaceTablet: tablet 6558265ff9e249c68123577958f1401a deleted and replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267
W20251023 20:15:17.865036 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2a27706ecdb4ab78bb96c10a9f6757f consensus state has no leader
W20251023 20:15:17.865931 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83da09ee0d10435ba89a19dc378d34eb consensus state has no leader
I20251023 20:15:17.874979 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet f1a6fe726ef844829a7ffec9d4494637 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:17.876682 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f7824cd559744860a83775305e47389c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.877333 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7824cd559744860a83775305e47389c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.879020 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_9a30e9c9272f47eb8fccd5e48ce47c30" } new_table_name: "default.Table_7b6d17b4f21740f0a5dadc35a23c919f" modify_external_catalogs: true
I20251023 20:15:17.881302 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_11159791d68d4dfb98fe2e8d7bb3afdc" } new_table_name: "default.Table_391ecae9d82d4a9089bc2d004ca92bc6" modify_external_catalogs: true
I20251023 20:15:17.882525 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.883513 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet f1a6fe726ef844829a7ffec9d4494637
I20251023 20:15:17.883966 23148 catalog_manager.cc:6719] ReplaceTablet: tablet f1a6fe726ef844829a7ffec9d4494637 deleted and replaced by tablet af899c01b4af46d39d27f984cc3d5940
I20251023 20:15:17.884599 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 955412de8bd1482ea7304872b48f38c0 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:17.887125 23790 tablet_bootstrap.cc:492] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.887669 23790 ts_tablet_manager.cc:1403] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.301s user 0.004s sys 0.004s
I20251023 20:15:17.888581 23790 raft_consensus.cc:359] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.889225 23790 raft_consensus.cc:385] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.889362 23790 raft_consensus.cc:740] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.890026 23790 consensus_queue.cc:260] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.890311 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:17.891465 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79e3d422a66f432789c0cf4a00e57af2 consensus state has no leader
I20251023 20:15:17.893354 23790 ts_tablet_manager.cc:1434] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:17.896124 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 6558265ff9e249c68123577958f1401a with delete_type TABLET_DATA_DELETED (replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:17.897881 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.898046 23794 raft_consensus.cc:493] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.898211 23794 raft_consensus.cc:515] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.898566 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.898705 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 1ae910ac41fd4e349f159ab27e44375a (DEFAULT_TABLE table=default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.898873 23794 leader_election.cc:290] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:17.899080 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e72922aae98430da968874f44603317 consensus state has no leader
I20251023 20:15:17.899329 23801 raft_consensus.cc:493] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.899356 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 (DEFAULT_TABLE table=default.Table_ee55921ad9e8422984b6afd71da0f4ee [id=c6c3d95bdfef4028a356f1365e9fa53d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.899516 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet 6558265ff9e249c68123577958f1401a with delete_type TABLET_DATA_DELETED (replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:17.899608 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e76f2e26dcd4cdfbf72de899f5a1458 consensus state has no leader
I20251023 20:15:17.899731 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e08dad1d0f643efb3a821a1b9196290. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.899734 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.900003 23790 tablet_bootstrap.cc:492] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:17.900104 23503 tablet_service.cc:1505] Processing CreateTablet for tablet 1ae910ac41fd4e349f159ab27e44375a (DEFAULT_TABLE table=default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.900429 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae910ac41fd4e349f159ab27e44375a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.900518 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae910ac41fd4e349f159ab27e44375a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.902007 23801 raft_consensus.cc:515] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.902555 23377 tablet_service.cc:1505] Processing CreateTablet for tablet d9f422575c7c4251879130501ab721fd (DEFAULT_TABLE table=default.Table_c01f9e9030974e2e8b43da4e68ef0145 [id=12899265e5544ac4837f4916f8bab077]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.903971 23783 ts_tablet_manager.cc:1950] T 73b4abd308054bac9d49b1d8e7dd2ebc P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:17.905798 23508 tablet_service.cc:1505] Processing CreateTablet for tablet d9f422575c7c4251879130501ab721fd (DEFAULT_TABLE table=default.Table_c01f9e9030974e2e8b43da4e68ef0145 [id=12899265e5544ac4837f4916f8bab077]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.905951 23801 leader_election.cc:290] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:17.906239 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f422575c7c4251879130501ab721fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.906610 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0232c33db59641b3af27bfd15cd7fb81" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:17.906809 23536 raft_consensus.cc:2468] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:17.907625 23604 leader_election.cc:304] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:17.908212 23801 raft_consensus.cc:2804] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:17.908299 23801 raft_consensus.cc:493] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:17.908453 23646 tablet_service.cc:1505] Processing CreateTablet for tablet d9f422575c7c4251879130501ab721fd (DEFAULT_TABLE table=default.Table_c01f9e9030974e2e8b43da4e68ef0145 [id=12899265e5544ac4837f4916f8bab077]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.908969 23801 raft_consensus.cc:3060] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:17.909770 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 6558265ff9e249c68123577958f1401a with delete_type TABLET_DATA_DELETED (replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.910099 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f422575c7c4251879130501ab721fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.911695 23785 ts_tablet_manager.cc:1929] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.912484 23941 raft_consensus.cc:493] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.918306 23801 raft_consensus.cc:515] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.912624 23941 raft_consensus.cc:515] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.920769 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:17.921006 23536 raft_consensus.cc:2468] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:17.921514 23604 leader_election.cc:304] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:17.921624 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4011bfe9c51f4264925601bcafccc515"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.921820 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e3186a2966f1433aa2cb5c751b84aa03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.922237 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4011bfe9c51f4264925601bcafccc515 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:17.922292 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3186a2966f1433aa2cb5c751b84aa03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.922655 23941 leader_election.cc:290] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:17.922896 23941 raft_consensus.cc:2804] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:17.923076 23941 raft_consensus.cc:493] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:17.923171 23941 raft_consensus.cc:3060] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:17.921626 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c7cdd3d9eaf8467ba67a5645d5e4547c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.923964 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7cdd3d9eaf8467ba67a5645d5e4547c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.925355 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0232c33db59641b3af27bfd15cd7fb81" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:17.925323 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:17.925539 23536 raft_consensus.cc:3060] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:17.926002 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.928133 23536 raft_consensus.cc:2468] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:17.912868 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0232c33db59641b3af27bfd15cd7fb81" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:17.928745 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:17.928936 23399 raft_consensus.cc:2468] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:17.929666 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:17.929728 23604 leader_election.cc:304] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:17.929896 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:17.930066 23942 raft_consensus.cc:2804] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:17.930013 23787 raft_consensus.cc:359] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.930434 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 73b4abd308054bac9d49b1d8e7dd2ebc (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
W20251023 20:15:17.930757 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 0438593fbbdf47cebfb64121acad104a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0438593fbbdf47cebfb64121acad104a already in progress: creating tablet
W20251023 20:15:17.930963 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6e1b4748c0244b258960b1025da47562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e1b4748c0244b258960b1025da47562 already in progress: creating tablet
W20251023 20:15:17.931156 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 86b9f4c5d9e0469c87071a33722ce3eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86b9f4c5d9e0469c87071a33722ce3eb already in progress: creating tablet
W20251023 20:15:17.931372 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ae388da9e4f94ba5af82a67088ab7097 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae388da9e4f94ba5af82a67088ab7097 already in progress: creating tablet
W20251023 20:15:17.931609 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f4778436aed64b53adb2bab54bcf263b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4778436aed64b53adb2bab54bcf263b already in progress: creating tablet
W20251023 20:15:17.931775 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 252b5480222f49439052c86a4115cc9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 252b5480222f49439052c86a4115cc9b already in progress: creating tablet
I20251023 20:15:17.931898 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0232c33db59641b3af27bfd15cd7fb81" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:17.932005 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 already in progress: creating tablet
I20251023 20:15:17.932094 23399 raft_consensus.cc:3060] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:17.934741 23399 raft_consensus.cc:2468] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:17.915825 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f422575c7c4251879130501ab721fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.937299 23943 raft_consensus.cc:493] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:17.936583 23801 leader_election.cc:290] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:17.937443 23943 raft_consensus.cc:515] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.938019 23941 raft_consensus.cc:515] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.938287 23943 leader_election.cc:290] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:17.938524 23670 raft_consensus.cc:2393] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
W20251023 20:15:17.939381 23605 leader_election.cc:343] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:17.939651 23941 leader_election.cc:290] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:17.940349 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:17.940495 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:17.940775 23536 raft_consensus.cc:3060] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:17.918758 23790 tablet_bootstrap.cc:654] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.942672 23784 ts_tablet_manager.cc:1929] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.942857 23784 log.cc:1199] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/50c75cce4b9d4c48800b2d0908676a14
I20251023 20:15:17.943291 23784 ts_tablet_manager.cc:1950] T 50c75cce4b9d4c48800b2d0908676a14 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:17.945142 23787 raft_consensus.cc:385] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.945259 23787 raft_consensus.cc:740] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:17.945367 23605 leader_election.cc:343] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:17.946602 23787 consensus_queue.cc:260] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.946709 23785 log.cc:1199] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/cc5ce157b6ca42da90b051a184983bfe
I20251023 20:15:17.947085 23785 ts_tablet_manager.cc:1950] T cc5ce157b6ca42da90b051a184983bfe P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:17.947127 23787 ts_tablet_manager.cc:1434] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.089s user 0.000s sys 0.002s
I20251023 20:15:17.947472 23787 tablet_bootstrap.cc:492] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:17.948051 23785 tablet_replica.cc:333] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:17.948271 23785 raft_consensus.cc:2243] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.948433 23785 raft_consensus.cc:2272] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.949023 23785 ts_tablet_manager.cc:1916] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.943406 23536 raft_consensus.cc:2468] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:17.950374 23604 leader_election.cc:304] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:17.950827 23941 raft_consensus.cc:2804] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:17.950974 23941 raft_consensus.cc:697] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:17.951321 23941 consensus_queue.cc:237] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:17.954604 23647 tablet_service.cc:1505] Processing CreateTablet for tablet 1ae910ac41fd4e349f159ab27e44375a (DEFAULT_TABLE table=default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.955014 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae910ac41fd4e349f159ab27e44375a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.956316 23784 tablet_replica.cc:333] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:17.959796 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet cc5ce157b6ca42da90b051a184983bfe (table default.Table_bd046e3e4fd9439c8aabbcee605207e4 [id=8b2cfa043c8147098454bfead6809936]) successfully deleted
I20251023 20:15:17.960120 23784 raft_consensus.cc:2243] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:17.960382 23784 raft_consensus.cc:2272] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:17.961200 23784 ts_tablet_manager.cc:1916] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:17.965917 23790 tablet_bootstrap.cc:492] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:17.966151 23790 ts_tablet_manager.cc:1403] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.005s
I20251023 20:15:17.967156 23790 raft_consensus.cc:359] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.967458 23790 raft_consensus.cc:385] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:17.967582 23790 raft_consensus.cc:740] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:17.967901 23790 consensus_queue.cc:260] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:17.970988 23790 ts_tablet_manager.cc:1434] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251023 20:15:17.958241 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:17.958218 23333 leader_election.cc:343] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: bd88285bf0a94fad858a25004609131c
W20251023 20:15:17.973652 23471 leader_election.cc:343] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:17.973804 23471 leader_election.cc:304] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:17.973765 23369 tablet_service.cc:1505] Processing CreateTablet for tablet b680b1cbc103403b91a1104248ecbdcf (DEFAULT_TABLE table=default.Table_0f114a37952f447ab986c5799d7ff3ca [id=6945386a3c8140c7b509b38c86326f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.973897 23784 ts_tablet_manager.cc:1929] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.974040 23784 log.cc:1199] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/5416da30c66645c19d36112980608527
I20251023 20:15:17.974071 23794 raft_consensus.cc:2749] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:17.974180 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b680b1cbc103403b91a1104248ecbdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.972681 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685213b588148febd6a23e1d2d6691d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:17.974582 23333 leader_election.cc:343] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: bd88285bf0a94fad858a25004609131c
I20251023 20:15:17.978554 23333 leader_election.cc:304] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:17.979146 23784 ts_tablet_manager.cc:1950] T 5416da30c66645c19d36112980608527 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:17.979429 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet 5416da30c66645c19d36112980608527 from {username='slave'} at 127.0.0.1:40384
W20251023 20:15:17.980052 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a708cd011802458aa0e2991fd4b4a2d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a708cd011802458aa0e2991fd4b4a2d3 already in progress: creating tablet
I20251023 20:15:17.978228 23790 tablet_bootstrap.cc:492] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:17.980345 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 66a2f3adb5d347a0adc198cd1ef93304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66a2f3adb5d347a0adc198cd1ef93304 already in progress: creating tablet
W20251023 20:15:17.980633 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 4ea63540149c4c02abb8ae51209060a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea63540149c4c02abb8ae51209060a3 already in progress: creating tablet
W20251023 20:15:17.980859 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6558265ff9e249c68123577958f1401a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6558265ff9e249c68123577958f1401a already in progress: creating tablet
I20251023 20:15:17.980875 23943 raft_consensus.cc:2749] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:17.981046 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f1a6fe726ef844829a7ffec9d4494637 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a6fe726ef844829a7ffec9d4494637 already in progress: creating tablet
I20251023 20:15:17.981570 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 50c75cce4b9d4c48800b2d0908676a14 (table default.Table_39878ca4314e43b8bf37b4cba1a5fb54 [id=c1b3327a41c8461cb83d9b31afa03e4d]) successfully deleted
I20251023 20:15:17.981855 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 5416da30c66645c19d36112980608527 (table default.Table_b6760837eeee4b5082522107b87d72f5 [id=6e38fec2eebd48b8a6364be16ec99997]) successfully deleted
I20251023 20:15:17.982897 23501 tablet_service.cc:1505] Processing CreateTablet for tablet b680b1cbc103403b91a1104248ecbdcf (DEFAULT_TABLE table=default.Table_0f114a37952f447ab986c5799d7ff3ca [id=6945386a3c8140c7b509b38c86326f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.983299 23790 tablet_bootstrap.cc:654] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.983322 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b680b1cbc103403b91a1104248ecbdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.993033 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e58c00cb57f34786864684583ab1adf6" } modify_external_catalogs: true
I20251023 20:15:17.993217 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e58c00cb57f34786864684583ab1adf6" } modify_external_catalogs: true
I20251023 20:15:17.966621 23787 tablet_bootstrap.cc:654] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:17.982606 23637 tablet_service.cc:1505] Processing CreateTablet for tablet b680b1cbc103403b91a1104248ecbdcf (DEFAULT_TABLE table=default.Table_0f114a37952f447ab986c5799d7ff3ca [id=6945386a3c8140c7b509b38c86326f5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.994995 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b680b1cbc103403b91a1104248ecbdcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.995213 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_bbd7b9cada174d2f8e03189510c6ea3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:17.995707 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbd7b9cada174d2f8e03189510c6ea3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:17.983672 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e31aebfc87734ffeb2c0b25c07cb3f70) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:17.999760 23784 tablet_replica.cc:333] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:17.999816 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 48436f3d8f1443db990a2d10b668b70a (DEFAULT_TABLE table=default.Table_7c03bf4f1b034b139999af6a349dccf0 [id=581d16998437467693ce84962d6717c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.000041 23784 raft_consensus.cc:2243] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.000201 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48436f3d8f1443db990a2d10b668b70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.000234 23784 raft_consensus.cc:2272] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.001305 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 994a555b24e4432a84601c54d422aa5d (DEFAULT_TABLE table=default.Table_c7cdd3d9eaf8467ba67a5645d5e4547c [id=dbdd5f771ae84b7498c203156aad3dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.001688 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994a555b24e4432a84601c54d422aa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.001693 23508 tablet_service.cc:1505] Processing CreateTablet for tablet f908f086946949829cac15418a584730 (DEFAULT_TABLE table=default.Table_a1c9b5357e504ba2a116c58ff86ad13c [id=25a11be83f484fa6abecb21ce1145e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.001983 23827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:18.002132 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f908f086946949829cac15418a584730. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.003003 23827 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251023 20:15:18.003293 23827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:18.005182 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 994a555b24e4432a84601c54d422aa5d (DEFAULT_TABLE table=default.Table_c7cdd3d9eaf8467ba67a5645d5e4547c [id=dbdd5f771ae84b7498c203156aad3dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.005522 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994a555b24e4432a84601c54d422aa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.007907 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 8e19f64a9ac44efba92928d0fa16819a
I20251023 20:15:18.011513 23784 ts_tablet_manager.cc:1916] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.015570 23508 tablet_service.cc:1505] Processing CreateTablet for tablet d1cd0a61827c49fd99dfd054c9a292f8 (DEFAULT_TABLE table=default.Table_f5ea72074b174da28b3738378167f49c [id=70e0efabed5a4dfe98eb7201a2b4aba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.016089 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1cd0a61827c49fd99dfd054c9a292f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.018229 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_36570bd0cbb4419b9e8daa3793ab2061"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:18.018318 23646 tablet_service.cc:1505] Processing CreateTablet for tablet d1cd0a61827c49fd99dfd054c9a292f8 (DEFAULT_TABLE table=default.Table_f5ea72074b174da28b3738378167f49c [id=70e0efabed5a4dfe98eb7201a2b4aba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.018694 23642 tablet_service.cc:1505] Processing CreateTablet for tablet b8e41ac34e84459581daef13914942b5 (DEFAULT_TABLE table=default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6 [id=77ff0c7e54e94defabffdbc82dbb609d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:18.018883 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36570bd0cbb4419b9e8daa3793ab2061 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.019038 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e41ac34e84459581daef13914942b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:18.019134 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ae388da9e4f94ba5af82a67088ab7097 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae388da9e4f94ba5af82a67088ab7097 already in progress: creating tablet
W20251023 20:15:18.019380 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 already in progress: creating tablet
W20251023 20:15:18.019635 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 4ea63540149c4c02abb8ae51209060a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea63540149c4c02abb8ae51209060a3 already in progress: creating tablet
I20251023 20:15:18.021118 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7e026bfaccc4482dba743310ed0adb4c" } modify_external_catalogs: true
I20251023 20:15:18.021390 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7e026bfaccc4482dba743310ed0adb4c" } modify_external_catalogs: true
I20251023 20:15:18.023314 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 2fea0afc8f63497d8b5cdf2352a20267 (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.024143 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fea0afc8f63497d8b5cdf2352a20267. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.027681 23787 tablet_bootstrap.cc:492] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.027885 23787 ts_tablet_manager.cc:1403] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.002s
I20251023 20:15:18.028703 23787 raft_consensus.cc:359] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.028945 23787 raft_consensus.cc:385] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.029104 23787 raft_consensus.cc:740] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.029918 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 523d72012c364457b9a2a299f2f789a1
I20251023 20:15:18.030560 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e19f64a9ac44efba92928d0fa16819a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.031062 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 994a555b24e4432a84601c54d422aa5d (DEFAULT_TABLE table=default.Table_c7cdd3d9eaf8467ba67a5645d5e4547c [id=dbdd5f771ae84b7498c203156aad3dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.031684 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994a555b24e4432a84601c54d422aa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.035663 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4e18a968658f470d870bc932102ebf00" } modify_external_catalogs: true
I20251023 20:15:18.035917 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4e18a968658f470d870bc932102ebf00" } modify_external_catalogs: true
I20251023 20:15:18.038115 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ca4eab8d09574f7abb17e56482816adf" } modify_external_catalogs: true
I20251023 20:15:18.038302 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ca4eab8d09574f7abb17e56482816adf" } modify_external_catalogs: true
I20251023 20:15:18.039278 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f7824cd559744860a83775305e47389c" } modify_external_catalogs: true
I20251023 20:15:18.039458 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f7824cd559744860a83775305e47389c" } modify_external_catalogs: true
I20251023 20:15:18.039628 23373 tablet_service.cc:1505] Processing CreateTablet for tablet af899c01b4af46d39d27f984cc3d5940 (DEFAULT_TABLE table=default.Table_20ccdc7f398e41459cbbb75ffa904e18 [id=8d5fb738eb684e27860e03c12802a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.040030 23787 consensus_queue.cc:260] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.040611 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af899c01b4af46d39d27f984cc3d5940. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.040659 23787 ts_tablet_manager.cc:1434] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20251023 20:15:18.040954 23787 tablet_bootstrap.cc:492] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.029429 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1cd0a61827c49fd99dfd054c9a292f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.029922 23648 tablet_service.cc:1505] Processing CreateTablet for tablet a636a4cdeb404e9f806d8e166ea57938 (DEFAULT_TABLE table=default.Table_4011bfe9c51f4264925601bcafccc515 [id=4b8e127e7d37433e8cfe495eda2bc815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.029951 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 48436f3d8f1443db990a2d10b668b70a (DEFAULT_TABLE table=default.Table_7c03bf4f1b034b139999af6a349dccf0 [id=581d16998437467693ce84962d6717c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.030126 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 600f0110f8bd4b05983d5922d871c266 (DEFAULT_TABLE table=default.Table_f7824cd559744860a83775305e47389c [id=4b9ecb1c28154b86bc2b6da54948416b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.030158 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (DEFAULT_TABLE table=default.Table_e3186a2966f1433aa2cb5c751b84aa03 [id=ec622f1a57064ff2a6ba1abb48b1181e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.030237 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 2fea0afc8f63497d8b5cdf2352a20267 (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.031062 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e19f64a9ac44efba92928d0fa16819a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.031215 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.042754 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fea0afc8f63497d8b5cdf2352a20267. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.042958 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48436f3d8f1443db990a2d10b668b70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.043704 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a636a4cdeb404e9f806d8e166ea57938. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.048511 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.048705 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600f0110f8bd4b05983d5922d871c266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.050479 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744d31fccbbe4fa59c56e24136e3b2b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:17.992476 23785 ts_tablet_manager.cc:1929] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:17.995792 23376 tablet_service.cc:1505] Processing CreateTablet for tablet f908f086946949829cac15418a584730 (DEFAULT_TABLE table=default.Table_a1c9b5357e504ba2a116c58ff86ad13c [id=25a11be83f484fa6abecb21ce1145e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.052882 23785 log.cc:1199] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d8ad21e3888a4915ae5f252c2fca64cb
W20251023 20:15:18.052947 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 523d72012c364457b9a2a299f2f789a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 523d72012c364457b9a2a299f2f789a1 already in progress: creating tablet
I20251023 20:15:18.053125 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f908f086946949829cac15418a584730. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:18.053257 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 8e19f64a9ac44efba92928d0fa16819a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e19f64a9ac44efba92928d0fa16819a already in progress: creating tablet
I20251023 20:15:18.053345 23785 ts_tablet_manager.cc:1950] T d8ad21e3888a4915ae5f252c2fca64cb P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.054385 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d8ad21e3888a4915ae5f252c2fca64cb (table default.Table_57a2a81ca5484113b5e2a0857d40dbbb [id=fc58c27a539f4454acbba8fbf3b62461]) successfully deleted
I20251023 20:15:17.995959 23368 tablet_service.cc:1505] Processing CreateTablet for tablet b8e41ac34e84459581daef13914942b5 (DEFAULT_TABLE table=default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6 [id=77ff0c7e54e94defabffdbc82dbb609d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.996140 23363 tablet_service.cc:1505] Processing CreateTablet for tablet d1cd0a61827c49fd99dfd054c9a292f8 (DEFAULT_TABLE table=default.Table_f5ea72074b174da28b3738378167f49c [id=70e0efabed5a4dfe98eb7201a2b4aba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.996551 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 600f0110f8bd4b05983d5922d871c266 (DEFAULT_TABLE table=default.Table_f7824cd559744860a83775305e47389c [id=4b9ecb1c28154b86bc2b6da54948416b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.997026 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (DEFAULT_TABLE table=default.Table_e3186a2966f1433aa2cb5c751b84aa03 [id=ec622f1a57064ff2a6ba1abb48b1181e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:17.997915 23362 tablet_service.cc:1505] Processing CreateTablet for tablet a636a4cdeb404e9f806d8e166ea57938 (DEFAULT_TABLE table=default.Table_4011bfe9c51f4264925601bcafccc515 [id=4b8e127e7d37433e8cfe495eda2bc815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.060382 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e41ac34e84459581daef13914942b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.061439 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a636a4cdeb404e9f806d8e166ea57938. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.032684 23644 tablet_service.cc:1505] Processing CreateTablet for tablet f908f086946949829cac15418a584730 (DEFAULT_TABLE table=default.Table_a1c9b5357e504ba2a116c58ff86ad13c [id=25a11be83f484fa6abecb21ce1145e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.063527 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f908f086946949829cac15418a584730. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.064003 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744d31fccbbe4fa59c56e24136e3b2b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.065140 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1cd0a61827c49fd99dfd054c9a292f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.067736 23790 tablet_bootstrap.cc:492] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.071959 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600f0110f8bd4b05983d5922d871c266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.037154 23942 raft_consensus.cc:697] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:18.072990 23942 consensus_queue.cc:237] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.073299 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet 2e338b982a564ea5b1b15c2146a96fa2 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.074011 23361 tablet_service.cc:1505] Processing CreateTablet for tablet f103bece2fa240c19c23ac51046c13ed (DEFAULT_TABLE table=default.Table_36570bd0cbb4419b9e8daa3793ab2061 [id=071ca3495096421eb75fea3fb71890ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.078150 23503 tablet_service.cc:1505] Processing CreateTablet for tablet b8e41ac34e84459581daef13914942b5 (DEFAULT_TABLE table=default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6 [id=77ff0c7e54e94defabffdbc82dbb609d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.078578 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e41ac34e84459581daef13914942b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.079000 23650 tablet_service.cc:1505] Processing CreateTablet for tablet af899c01b4af46d39d27f984cc3d5940 (DEFAULT_TABLE table=default.Table_20ccdc7f398e41459cbbb75ffa904e18 [id=8d5fb738eb684e27860e03c12802a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.079536 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a8ba7ab28dfa4ac38e0b58ba2115495c
I20251023 20:15:18.079986 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d943501e515b41e6a00142359321f818"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.080503 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d943501e515b41e6a00142359321f818 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.080637 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 600f0110f8bd4b05983d5922d871c266
I20251023 20:15:18.082790 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.084136 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_31ff42d4292e455c90c8ad235f2154fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.084657 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31ff42d4292e455c90c8ad235f2154fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.084810 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_664e281e682d4426b64cb60b0af1ac18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.085287 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_664e281e682d4426b64cb60b0af1ac18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.085366 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.085876 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ff93d93fcfce4bd686e1e6fa95a31ecc" } new_table_name: "default.Table_014b7052f2aa423e8f841e4be8da3a9c" modify_external_catalogs: true
W20251023 20:15:18.085999 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 600f0110f8bd4b05983d5922d871c266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600f0110f8bd4b05983d5922d871c266 already in progress: creating tablet
I20251023 20:15:18.086273 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:18.086817 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a8ba7ab28dfa4ac38e0b58ba2115495c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ba7ab28dfa4ac38e0b58ba2115495c already in progress: creating tablet
I20251023 20:15:18.086903 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ea99b2e89c224754a8eba8255433cd14" } new_table_name: "default.Table_b90b24a3a90b47129da870a81b32f822" modify_external_catalogs: true
I20251023 20:15:18.088723 23786 tablet_bootstrap.cc:492] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:18.091092 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca01bea2bb146ca8797a9592a840a81 (DEFAULT_TABLE table=default.Table_bbd7b9cada174d2f8e03189510c6ea3d [id=39f5bef03d6c49b0ba2af0223cb9e573]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.092276 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet cff337a6e728489b99fa8c5d4dfece2b
I20251023 20:15:18.093954 23790 ts_tablet_manager.cc:1403] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.004s
I20251023 20:15:18.094077 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.094076 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.093988 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f103bece2fa240c19c23ac51046c13ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.094717 23790 raft_consensus.cc:359] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.094859 23790 raft_consensus.cc:385] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.094915 23790 raft_consensus.cc:740] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.095108 23790 consensus_queue.cc:260] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.095490 23790 ts_tablet_manager.cc:1434] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:18.095800 23790 tablet_bootstrap.cc:492] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.097215 23785 tablet_replica.cc:333] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.099159 23785 raft_consensus.cc:2243] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.099361 23785 raft_consensus.cc:2272] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.100214 23785 ts_tablet_manager.cc:1916] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.100507 23502 tablet_service.cc:1505] Processing CreateTablet for tablet f103bece2fa240c19c23ac51046c13ed (DEFAULT_TABLE table=default.Table_36570bd0cbb4419b9e8daa3793ab2061 [id=071ca3495096421eb75fea3fb71890ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.100564 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.100734 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 600f0110f8bd4b05983d5922d871c266 (DEFAULT_TABLE table=default.Table_f7824cd559744860a83775305e47389c [id=4b9ecb1c28154b86bc2b6da54948416b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.101186 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.101572 23509 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.101725 23500 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.101830 23511 tablet_service.cc:1505] Processing CreateTablet for tablet af899c01b4af46d39d27f984cc3d5940 (DEFAULT_TABLE table=default.Table_20ccdc7f398e41459cbbb75ffa904e18 [id=8d5fb738eb684e27860e03c12802a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.102346 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e19f64a9ac44efba92928d0fa16819a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.105484 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af899c01b4af46d39d27f984cc3d5940. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:18.106786 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cff337a6e728489b99fa8c5d4dfece2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cff337a6e728489b99fa8c5d4dfece2b already in progress: creating tablet
W20251023 20:15:18.108418 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa794b3bec4b4b3993c814ea7cd02ee2 consensus state has no leader
W20251023 20:15:18.108942 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c317dc31cb0448d985512e6ba98735be consensus state has no leader
I20251023 20:15:18.108980 23785 ts_tablet_manager.cc:1929] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.109125 23785 log.cc:1199] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/dc8f615c621043da88d52f3964b0a0ec
I20251023 20:15:18.109457 23785 ts_tablet_manager.cc:1950] T dc8f615c621043da88d52f3964b0a0ec P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.110287 23785 tablet_replica.cc:333] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.110478 23785 raft_consensus.cc:2243] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.110641 23785 raft_consensus.cc:2272] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.110917 23790 tablet_bootstrap.cc:654] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.111166 23785 ts_tablet_manager.cc:1916] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.113023 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 2fea0afc8f63497d8b5cdf2352a20267 (DEFAULT_TABLE table=default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.113858 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 50c75cce4b9d4c48800b2d0908676a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.114002 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 48436f3d8f1443db990a2d10b668b70a (DEFAULT_TABLE table=default.Table_7c03bf4f1b034b139999af6a349dccf0 [id=581d16998437467693ce84962d6717c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.114130 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet dc8f615c621043da88d52f3964b0a0ec (table default.Table_4fba73266b6d4d36958e8f0566eb624e [id=d7983b16809245a2a73b2a6cd79f6e29]) successfully deleted
I20251023 20:15:18.114521 23510 tablet_service.cc:1505] Processing CreateTablet for tablet a636a4cdeb404e9f806d8e166ea57938 (DEFAULT_TABLE table=default.Table_4011bfe9c51f4264925601bcafccc515 [id=4b8e127e7d37433e8cfe495eda2bc815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.115022 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_07cf9b48edc541d7999a884181db7e0d" } new_table_name: "default.Table_e446e75e73024a1fb40206c4a4c835ac" modify_external_catalogs: true
I20251023 20:15:18.115085 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_36570bd0cbb4419b9e8daa3793ab2061" } new_table_name: "default.Table_c7fa864846ef4acbba390d7f493a8b58" modify_external_catalogs: true
I20251023 20:15:18.115449 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_aba84ed1ff684cc690f31c5615ec1700"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.115931 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aba84ed1ff684cc690f31c5615ec1700 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.116523 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4011bfe9c51f4264925601bcafccc515" } new_table_name: "default.Table_96e1782778fe489e8919612f34dc710f" modify_external_catalogs: true
I20251023 20:15:18.117692 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.118180 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.120559 23786 ts_tablet_manager.cc:1403] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.314s user 0.003s sys 0.002s
I20251023 20:15:18.121388 23786 raft_consensus.cc:359] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.121606 23786 raft_consensus.cc:385] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.121691 23786 raft_consensus.cc:740] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.121928 23786 consensus_queue.cc:260] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.122413 23786 ts_tablet_manager.cc:1434] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:18.122546 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f103bece2fa240c19c23ac51046c13ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.126233 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48436f3d8f1443db990a2d10b668b70a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:18.127750 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f103bece2fa240c19c23ac51046c13ed consensus state has no leader
I20251023 20:15:18.129264 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fea0afc8f63497d8b5cdf2352a20267. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:18.131651 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d58bceecba442898a3d742fa49a4a2 consensus state has no leader
I20251023 20:15:18.133255 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a636a4cdeb404e9f806d8e166ea57938. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.135825 23785 ts_tablet_manager.cc:1929] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.136061 23785 log.cc:1199] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/bd88285bf0a94fad858a25004609131c
I20251023 20:15:18.136437 23785 ts_tablet_manager.cc:1950] T bd88285bf0a94fad858a25004609131c P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.138295 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet bd88285bf0a94fad858a25004609131c (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]) successfully deleted
W20251023 20:15:18.140707 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 86b9f4c5d9e0469c87071a33722ce3eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86b9f4c5d9e0469c87071a33722ce3eb already in progress: creating tablet
W20251023 20:15:18.141026 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet ae388da9e4f94ba5af82a67088ab7097 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae388da9e4f94ba5af82a67088ab7097 already in progress: creating tablet
W20251023 20:15:18.141413 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet f4778436aed64b53adb2bab54bcf263b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4778436aed64b53adb2bab54bcf263b already in progress: creating tablet
I20251023 20:15:18.141558 23786 tablet_bootstrap.cc:492] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:18.141743 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 already in progress: creating tablet
I20251023 20:15:18.143776 23786 tablet_bootstrap.cc:654] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:18.144240 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 252b5480222f49439052c86a4115cc9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 252b5480222f49439052c86a4115cc9b already in progress: creating tablet
I20251023 20:15:18.144259 23785 tablet_replica.cc:333] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.144459 23785 raft_consensus.cc:2243] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:18.144508 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 4ea63540149c4c02abb8ae51209060a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea63540149c4c02abb8ae51209060a3 already in progress: creating tablet
I20251023 20:15:18.144672 23785 raft_consensus.cc:2272] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:18.144886 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 66a2f3adb5d347a0adc198cd1ef93304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66a2f3adb5d347a0adc198cd1ef93304 already in progress: creating tablet
I20251023 20:15:18.145816 23790 tablet_bootstrap.cc:492] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.145972 23790 ts_tablet_manager.cc:1403] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.018s
W20251023 20:15:18.148288 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a708cd011802458aa0e2991fd4b4a2d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a708cd011802458aa0e2991fd4b4a2d3 already in progress: creating tablet
I20251023 20:15:18.148382 23790 raft_consensus.cc:359] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.148607 23790 raft_consensus.cc:385] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:18.148701 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 6558265ff9e249c68123577958f1401a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6558265ff9e249c68123577958f1401a already in progress: creating tablet
I20251023 20:15:18.148777 23790 raft_consensus.cc:740] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:18.149116 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet f1a6fe726ef844829a7ffec9d4494637 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a6fe726ef844829a7ffec9d4494637 already in progress: creating tablet
I20251023 20:15:18.147588 23785 ts_tablet_manager.cc:1916] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.149840 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e9fdfa4d73c241e8a7d041b6b327d445"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:18.151834 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.151765 23790 consensus_queue.cc:260] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:18.152274 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a636a4cdeb404e9f806d8e166ea57938 consensus state has no leader
I20251023 20:15:18.152426 23790 ts_tablet_manager.cc:1434] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20251023 20:15:18.152633 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.153069 23790 tablet_bootstrap.cc:492] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.157635 23790 tablet_bootstrap.cc:654] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.161141 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600f0110f8bd4b05983d5922d871c266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.162093 23785 ts_tablet_manager.cc:1929] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.162287 23785 log.cc:1199] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/1685213b588148febd6a23e1d2d6691d
I20251023 20:15:18.162720 23785 ts_tablet_manager.cc:1950] T 1685213b588148febd6a23e1d2d6691d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.163635 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f4de4bcad2f34502a1b63889f5b5e58a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.164330 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4de4bcad2f34502a1b63889f5b5e58a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.164772 23790 tablet_bootstrap.cc:492] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.164716 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_52af8fa4ff964f16b6fe2b43e94dddd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:18.165115 23790 ts_tablet_manager.cc:1403] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
W20251023 20:15:18.165151 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52af8fa4ff964f16b6fe2b43e94dddd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.165970 23790 raft_consensus.cc:359] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.164465 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 1685213b588148febd6a23e1d2d6691d (table default.Table_2e1bf7e58ab84283b5f61b77865778ca [id=580a286a829e44f9b9992f3b0b7482eb]) successfully deleted
I20251023 20:15:18.163661 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7250290cab0942a3b36660561c7b8e0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.166478 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7250290cab0942a3b36660561c7b8e0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.166325 23790 raft_consensus.cc:385] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.167817 23790 raft_consensus.cc:740] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.168220 23790 consensus_queue.cc:260] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.168828 23790 ts_tablet_manager.cc:1434] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251023 20:15:18.169202 23790 tablet_bootstrap.cc:492] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.170518 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (DEFAULT_TABLE table=default.Table_e3186a2966f1433aa2cb5c751b84aa03 [id=ec622f1a57064ff2a6ba1abb48b1181e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.170974 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744d31fccbbe4fa59c56e24136e3b2b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.171476 23790 tablet_bootstrap.cc:654] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.164160 23785 tablet_replica.cc:333] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.172781 23785 raft_consensus.cc:2243] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.173028 23785 raft_consensus.cc:2272] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.173859 23785 ts_tablet_manager.cc:1916] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.179682 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 949e31d4a9ee4579843ed54153e22c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 9c0aeac2f20c4ca48c130a0c851f290f) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.181254 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af899c01b4af46d39d27f984cc3d5940. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.181953 23785 ts_tablet_manager.cc:1929] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.182725 23785 log.cc:1199] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/02b79ba69a1c4f8ebc7ecb795ac0d1d5
I20251023 20:15:18.183229 23785 ts_tablet_manager.cc:1950] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.184841 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca01bea2bb146ca8797a9592a840a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.187108 23785 tablet_replica.cc:333] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.187475 23785 raft_consensus.cc:2243] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.187875 23785 raft_consensus.cc:2272] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.188846 23785 ts_tablet_manager.cc:1916] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.193305 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
W20251023 20:15:18.193717 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 8e19f64a9ac44efba92928d0fa16819a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e19f64a9ac44efba92928d0fa16819a already in progress: creating tablet
I20251023 20:15:18.193873 23790 tablet_bootstrap.cc:492] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.194031 23790 ts_tablet_manager.cc:1403] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.006s
W20251023 20:15:18.194170 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 523d72012c364457b9a2a299f2f789a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 523d72012c364457b9a2a299f2f789a1 already in progress: creating tablet
W20251023 20:15:18.194860 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 600f0110f8bd4b05983d5922d871c266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600f0110f8bd4b05983d5922d871c266 already in progress: creating tablet
W20251023 20:15:18.195174 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet cff337a6e728489b99fa8c5d4dfece2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cff337a6e728489b99fa8c5d4dfece2b already in progress: creating tablet
W20251023 20:15:18.195374 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a8ba7ab28dfa4ac38e0b58ba2115495c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ba7ab28dfa4ac38e0b58ba2115495c already in progress: creating tablet
I20251023 20:15:18.197235 23787 tablet_bootstrap.cc:654] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.194854 23790 raft_consensus.cc:359] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.202215 23790 raft_consensus.cc:385] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.202379 23790 raft_consensus.cc:740] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.202733 23790 consensus_queue.cc:260] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.203480 23790 ts_tablet_manager.cc:1434] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251023 20:15:18.204185 23790 tablet_bootstrap.cc:492] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.206239 23790 tablet_bootstrap.cc:654] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.210291 23785 ts_tablet_manager.cc:1929] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.211382 23785 log.cc:1199] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/50c75cce4b9d4c48800b2d0908676a14
I20251023 20:15:18.211916 23785 ts_tablet_manager.cc:1950] T 50c75cce4b9d4c48800b2d0908676a14 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.214082 23785 tablet_replica.cc:333] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.214583 23785 raft_consensus.cc:2243] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.215062 23785 raft_consensus.cc:2272] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.215857 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca01bea2bb146ca8797a9592a840a81 (DEFAULT_TABLE table=default.Table_bbd7b9cada174d2f8e03189510c6ea3d [id=39f5bef03d6c49b0ba2af0223cb9e573]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.216135 23790 tablet_bootstrap.cc:492] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.216327 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca01bea2bb146ca8797a9592a840a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.216380 23790 ts_tablet_manager.cc:1403] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20251023 20:15:18.217367 23790 raft_consensus.cc:359] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.217643 23790 raft_consensus.cc:385] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.217746 23790 raft_consensus.cc:740] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.218045 23790 consensus_queue.cc:260] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.218714 23790 ts_tablet_manager.cc:1434] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:18.219046 23790 tablet_bootstrap.cc:492] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.220886 23785 ts_tablet_manager.cc:1916] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.222666 23786 tablet_bootstrap.cc:492] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:18.221993 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 5416da30c66645c19d36112980608527 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.222818 23786 ts_tablet_manager.cc:1403] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.001s
I20251023 20:15:18.222074 23790 tablet_bootstrap.cc:654] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.223420 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 50c75cce4b9d4c48800b2d0908676a14 (table default.Table_39878ca4314e43b8bf37b4cba1a5fb54 [id=c1b3327a41c8461cb83d9b31afa03e4d]) successfully deleted
I20251023 20:15:18.223609 23786 raft_consensus.cc:359] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.223790 23786 raft_consensus.cc:385] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.223877 23786 raft_consensus.cc:740] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.224066 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet 83da09ee0d10435ba89a19dc378d34eb from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.224121 23786 consensus_queue.cc:260] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.224630 23786 ts_tablet_manager.cc:1434] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:18.224989 23786 tablet_bootstrap.cc:492] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:18.224152 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 0ca01bea2bb146ca8797a9592a840a81 (DEFAULT_TABLE table=default.Table_bbd7b9cada174d2f8e03189510c6ea3d [id=39f5bef03d6c49b0ba2af0223cb9e573]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.227665 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca01bea2bb146ca8797a9592a840a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.229298 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 1685213b588148febd6a23e1d2d6691d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.231168 23784 ts_tablet_manager.cc:1929] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.231323 23784 log.cc:1199] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e44dfb4fd3d04a58b5fcc6102e055ff1
I20251023 20:15:18.231734 23784 ts_tablet_manager.cc:1950] T e44dfb4fd3d04a58b5fcc6102e055ff1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:18.232280 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet e44dfb4fd3d04a58b5fcc6102e055ff1 (table default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:18.233323 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:18.233785 23142 catalog_manager.cc:6719] ReplaceTablet: tablet 83da09ee0d10435ba89a19dc378d34eb deleted and replaced by tablet 25079532547d4175802b84877790662d
I20251023 20:15:18.234624 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:18.235318 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 83da09ee0d10435ba89a19dc378d34eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83da09ee0d10435ba89a19dc378d34eb already in progress: creating tablet
I20251023 20:15:18.236411 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.239490 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_58731fce9c224ff0b59d4b241110e518" } modify_external_catalogs: true
I20251023 20:15:18.239728 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_58731fce9c224ff0b59d4b241110e518" } modify_external_catalogs: true
I20251023 20:15:18.240703 23785 ts_tablet_manager.cc:1929] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:18.240867 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 252b5480222f49439052c86a4115cc9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 252b5480222f49439052c86a4115cc9b already in progress: creating tablet
I20251023 20:15:18.242585 23786 tablet_bootstrap.cc:654] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.242920 23790 tablet_bootstrap.cc:492] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:18.243110 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f4778436aed64b53adb2bab54bcf263b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4778436aed64b53adb2bab54bcf263b already in progress: creating tablet
I20251023 20:15:18.243181 23790 ts_tablet_manager.cc:1403] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
W20251023 20:15:18.243736 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 66a2f3adb5d347a0adc198cd1ef93304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66a2f3adb5d347a0adc198cd1ef93304 already in progress: creating tablet
I20251023 20:15:18.244066 23790 raft_consensus.cc:359] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.244321 23790 raft_consensus.cc:385] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.244446 23790 raft_consensus.cc:740] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.244760 23790 consensus_queue.cc:260] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.245718 23790 ts_tablet_manager.cc:1434] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:18.246068 23790 tablet_bootstrap.cc:492] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.248448 23790 tablet_bootstrap.cc:654] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.249248 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 8da82e6d24b14a129b91bb5298139d07
W20251023 20:15:18.249864 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a708cd011802458aa0e2991fd4b4a2d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a708cd011802458aa0e2991fd4b4a2d3 already in progress: creating tablet
I20251023 20:15:18.248644 23785 log.cc:1199] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/949e31d4a9ee4579843ed54153e22c7d
I20251023 20:15:18.250603 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.250711 23785 ts_tablet_manager.cc:1950] T 949e31d4a9ee4579843ed54153e22c7d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:18.251281 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 8da82e6d24b14a129b91bb5298139d07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8da82e6d24b14a129b91bb5298139d07 already in progress: creating tablet
I20251023 20:15:18.251842 23785 tablet_replica.cc:333] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.251716 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_eaed585bf4154e50a3bbf4fc798dcd12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.251957 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f1a6fe726ef844829a7ffec9d4494637 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a6fe726ef844829a7ffec9d4494637 already in progress: creating tablet
I20251023 20:15:18.252151 23785 raft_consensus.cc:2243] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:18.252283 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaed585bf4154e50a3bbf4fc798dcd12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.252375 23785 raft_consensus.cc:2272] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:18.252978 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6558265ff9e249c68123577958f1401a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6558265ff9e249c68123577958f1401a already in progress: creating tablet
I20251023 20:15:18.253012 23785 ts_tablet_manager.cc:1916] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:18.255923 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 600f0110f8bd4b05983d5922d871c266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600f0110f8bd4b05983d5922d871c266 already in progress: creating tablet
W20251023 20:15:18.257231 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cff337a6e728489b99fa8c5d4dfece2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cff337a6e728489b99fa8c5d4dfece2b already in progress: creating tablet
I20251023 20:15:18.259749 23785 ts_tablet_manager.cc:1929] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.250784 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:18.260576 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 523d72012c364457b9a2a299f2f789a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 523d72012c364457b9a2a299f2f789a1 already in progress: creating tablet
W20251023 20:15:18.261193 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 8e19f64a9ac44efba92928d0fa16819a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e19f64a9ac44efba92928d0fa16819a already in progress: creating tablet
I20251023 20:15:18.261644 23790 tablet_bootstrap.cc:492] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.261931 23790 ts_tablet_manager.cc:1403] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20251023 20:15:18.262948 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet 5964da70989b4e6cb77fe5d0a52ce9cf from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.262874 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ce123b77e83b41c4bf1fc149512c47d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.263649 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce123b77e83b41c4bf1fc149512c47d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.264122 23787 tablet_bootstrap.cc:492] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.264293 23787 ts_tablet_manager.cc:1403] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.223s user 0.000s sys 0.003s
I20251023 20:15:18.264907 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7250290cab0942a3b36660561c7b8e0c" } modify_external_catalogs: true
I20251023 20:15:18.265112 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7250290cab0942a3b36660561c7b8e0c" } modify_external_catalogs: true
I20251023 20:15:18.265064 23787 raft_consensus.cc:359] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.265275 23787 raft_consensus.cc:385] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.265856 23787 raft_consensus.cc:740] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.266089 23787 consensus_queue.cc:260] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.266551 23787 ts_tablet_manager.cc:1434] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:18.266629 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.266901 23787 tablet_bootstrap.cc:492] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.267824 23790 raft_consensus.cc:359] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.268141 23790 raft_consensus.cc:385] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.268249 23790 raft_consensus.cc:740] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.268580 23790 consensus_queue.cc:260] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.269320 23790 ts_tablet_manager.cc:1434] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251023 20:15:18.269747 23790 tablet_bootstrap.cc:492] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.270968 23646 tablet_service.cc:1505] Processing CreateTablet for tablet f103bece2fa240c19c23ac51046c13ed (DEFAULT_TABLE table=default.Table_36570bd0cbb4419b9e8daa3793ab2061 [id=071ca3495096421eb75fea3fb71890ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.272596 23790 tablet_bootstrap.cc:654] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.273504 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:18.274091 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a8ba7ab28dfa4ac38e0b58ba2115495c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ba7ab28dfa4ac38e0b58ba2115495c already in progress: creating tablet
I20251023 20:15:18.274423 23785 log.cc:1199] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/5416da30c66645c19d36112980608527
I20251023 20:15:18.274520 23784 tablet_replica.cc:333] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:18.274866 23784 raft_consensus.cc:2243] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.274943 23785 ts_tablet_manager.cc:1950] T 5416da30c66645c19d36112980608527 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.275104 23784 raft_consensus.cc:2272] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.275362 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 949e31d4a9ee4579843ed54153e22c7d (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]) successfully deleted
I20251023 20:15:18.275905 23784 ts_tablet_manager.cc:1916] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.280839 23790 tablet_bootstrap.cc:492] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.281105 23790 ts_tablet_manager.cc:1403] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20251023 20:15:18.282076 23790 raft_consensus.cc:359] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.282452 23790 raft_consensus.cc:385] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.282599 23790 raft_consensus.cc:740] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.282886 23790 consensus_queue.cc:260] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.283538 23790 ts_tablet_manager.cc:1434] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:18.283874 23790 tablet_bootstrap.cc:492] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.285842 23790 tablet_bootstrap.cc:654] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.281883 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 5964da70989b4e6cb77fe5d0a52ce9cf
I20251023 20:15:18.287837 23144 catalog_manager.cc:6719] ReplaceTablet: tablet 5964da70989b4e6cb77fe5d0a52ce9cf deleted and replaced by tablet 00ec606cbf834ccd8762e034b520e58e
I20251023 20:15:18.288460 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.288846 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.288983 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 73b4abd308054bac9d49b1d8e7dd2ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.289273 23785 tablet_replica.cc:333] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.289482 23785 raft_consensus.cc:2243] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.289665 23785 raft_consensus.cc:2272] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:18.289624 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5964da70989b4e6cb77fe5d0a52ce9cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5964da70989b4e6cb77fe5d0a52ce9cf already in progress: creating tablet
I20251023 20:15:18.290380 23785 ts_tablet_manager.cc:1916] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.293272 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.296511 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet 86b9f4c5d9e0469c87071a33722ce3eb from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.298719 23785 ts_tablet_manager.cc:1929] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.298938 23785 log.cc:1199] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/73b4abd308054bac9d49b1d8e7dd2ebc
I20251023 20:15:18.299403 23785 ts_tablet_manager.cc:1950] T 73b4abd308054bac9d49b1d8e7dd2ebc P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.303040 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f103bece2fa240c19c23ac51046c13ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.305781 23790 tablet_bootstrap.cc:492] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.306039 23790 ts_tablet_manager.cc:1403] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20251023 20:15:18.306716 23148 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 7e9baf7bd83e438dbb1e3f9b4be1f3b0
I20251023 20:15:18.307330 23790 raft_consensus.cc:359] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.307662 23790 raft_consensus.cc:385] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.307778 23790 raft_consensus.cc:740] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.308106 23790 consensus_queue.cc:260] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.309053 23790 ts_tablet_manager.cc:1434] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:18.309454 23790 tablet_bootstrap.cc:492] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.311293 23790 tablet_bootstrap.cc:654] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.314284 23786 tablet_bootstrap.cc:492] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:18.314610 23786 ts_tablet_manager.cc:1403] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20251023 20:15:18.315354 23786 raft_consensus.cc:359] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.315496 23787 tablet_bootstrap.cc:654] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.315693 23786 raft_consensus.cc:385] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.315786 23786 raft_consensus.cc:740] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.317481 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 5416da30c66645c19d36112980608527 (table default.Table_b6760837eeee4b5082522107b87d72f5 [id=6e38fec2eebd48b8a6364be16ec99997]) successfully deleted
I20251023 20:15:18.318058 23784 ts_tablet_manager.cc:1929] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.318202 23784 log.cc:1199] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/73b4abd308054bac9d49b1d8e7dd2ebc
I20251023 20:15:18.318030 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet 283166bbd8004ce3bca9f2ecbdbcbf48 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.318640 23784 ts_tablet_manager.cc:1950] T 73b4abd308054bac9d49b1d8e7dd2ebc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:18.318629 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 83da09ee0d10435ba89a19dc378d34eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83da09ee0d10435ba89a19dc378d34eb already in progress: creating tablet
W20251023 20:15:18.318849 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 8da82e6d24b14a129b91bb5298139d07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8da82e6d24b14a129b91bb5298139d07 already in progress: creating tablet
I20251023 20:15:18.319090 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 73b4abd308054bac9d49b1d8e7dd2ebc (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:18.318910 23786 consensus_queue.cc:260] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:18.319312 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 5964da70989b4e6cb77fe5d0a52ce9cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5964da70989b4e6cb77fe5d0a52ce9cf already in progress: creating tablet
I20251023 20:15:18.320025 23786 ts_tablet_manager.cc:1434] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:18.325647 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ee55921ad9e8422984b6afd71da0f4ee" } modify_external_catalogs: true
I20251023 20:15:18.325852 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ee55921ad9e8422984b6afd71da0f4ee" } modify_external_catalogs: true
I20251023 20:15:18.326471 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a1c9b5357e504ba2a116c58ff86ad13c" } modify_external_catalogs: true
I20251023 20:15:18.326630 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a1c9b5357e504ba2a116c58ff86ad13c" } modify_external_catalogs: true
I20251023 20:15:18.327545 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_eaed585bf4154e50a3bbf4fc798dcd12" } modify_external_catalogs: true
I20251023 20:15:18.327711 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_eaed585bf4154e50a3bbf4fc798dcd12" } modify_external_catalogs: true
I20251023 20:15:18.328084 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_31ff42d4292e455c90c8ad235f2154fa" } new_table_name: "default.Table_fa3c4b34d4744caf8f1fe02cf5b2326c" modify_external_catalogs: true
I20251023 20:15:18.328981 23790 tablet_bootstrap.cc:492] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.329450 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_9899db7c169244f38fdb46b1fd263e1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:18.329651 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_187505c662504914a73c39146522f45e" } new_table_name: "default.Table_fc18b4fbf71a45f6b3cfee31effd8eef" modify_external_catalogs: true
W20251023 20:15:18.329993 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9899db7c169244f38fdb46b1fd263e1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.331404 23790 ts_tablet_manager.cc:1403] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.003s
I20251023 20:15:18.332260 23790 raft_consensus.cc:359] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.332494 23790 raft_consensus.cc:385] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.332618 23790 raft_consensus.cc:740] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.332877 23790 consensus_queue.cc:260] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.333611 23790 ts_tablet_manager.cc:1434] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:18.333956 23790 tablet_bootstrap.cc:492] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.336331 23790 tablet_bootstrap.cc:654] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.338892 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 73b4abd308054bac9d49b1d8e7dd2ebc (table default.Table_2741dc8a0a2845cc945ac27e11d1bb98 [id=9b4940d58a0a4b81bd688d4f70567ef7]) successfully deleted
I20251023 20:15:18.339308 23784 tablet_replica.cc:333] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:18.339922 23784 raft_consensus.cc:2243] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.340148 23784 raft_consensus.cc:2272] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.339733 23786 tablet_bootstrap.cc:492] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:18.341892 23784 ts_tablet_manager.cc:1916] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.345726 23786 tablet_bootstrap.cc:654] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.349479 23790 tablet_bootstrap.cc:492] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.349829 23790 ts_tablet_manager.cc:1403] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20251023 20:15:18.350638 23790 raft_consensus.cc:359] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.350862 23790 raft_consensus.cc:385] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.350942 23790 raft_consensus.cc:740] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.351205 23790 consensus_queue.cc:260] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.351974 23790 ts_tablet_manager.cc:1434] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251023 20:15:18.352285 23790 tablet_bootstrap.cc:492] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.353928 23784 ts_tablet_manager.cc:1929] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.354419 23790 tablet_bootstrap.cc:654] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.355569 23784 log.cc:1199] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/1685213b588148febd6a23e1d2d6691d
I20251023 20:15:18.356150 23784 ts_tablet_manager.cc:1950] T 1685213b588148febd6a23e1d2d6691d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:18.356251 23943 raft_consensus.cc:493] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.356392 23943 raft_consensus.cc:515] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.356469 23942 raft_consensus.cc:493] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.356639 23942 raft_consensus.cc:515] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.356947 23943 leader_election.cc:290] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:18.357275 23942 leader_election.cc:290] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:18.358166 23146 catalog_manager.cc:5649] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:18.358588 23146 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998]). Expected version 4 got 0
I20251023 20:15:18.359576 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:18.361275 23605 leader_election.cc:343] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 02b79ba69a1c4f8ebc7ecb795ac0d1d5
W20251023 20:15:18.362213 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a21ec49194e24723b31fd9c86f1f85e4 consensus state has no leader
W20251023 20:15:18.362439 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0150ab3778c34c77a7e772dd6e82b32f consensus state has no leader
I20251023 20:15:18.362655 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_57771b43db0747d68d0dea8e558351e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:18.362630 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_24eda21d04d848c7b8971d30ac29f9f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:18.362792 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d07400a150ac4a6c86bac1caeebe8739" } new_table_name: "default.Table_aab23aa8d50a4902b09495b3fa1ea6ca" modify_external_catalogs: true
W20251023 20:15:18.363260 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24eda21d04d848c7b8971d30ac29f9f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.364109 23142 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
W20251023 20:15:18.364451 23604 leader_election.cc:343] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 02b79ba69a1c4f8ebc7ecb795ac0d1d5
I20251023 20:15:18.364583 23604 leader_election.cc:304] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:18.363262 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57771b43db0747d68d0dea8e558351e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.365453 23144 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 514a44eb4fab4045807f60e17f9b89a3
I20251023 20:15:18.365458 23790 tablet_bootstrap.cc:492] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.365615 23790 ts_tablet_manager.cc:1403] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20251023 20:15:18.365770 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_1fc3a0e0fe484168b2f767fdcc3eba64" } new_table_name: "default.Table_83bf80c4c34c4312be19e260564d4916" modify_external_catalogs: true
I20251023 20:15:18.366017 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b93a09969d41431aa77623ab62d6f6e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.366474 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b93a09969d41431aa77623ab62d6f6e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.366431 23790 raft_consensus.cc:359] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.366617 23790 raft_consensus.cc:385] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.366701 23790 raft_consensus.cc:740] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.366741 23787 tablet_bootstrap.cc:492] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.366880 23787 ts_tablet_manager.cc:1403] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.100s user 0.000s sys 0.003s
I20251023 20:15:18.366909 23790 consensus_queue.cc:260] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.367683 23942 raft_consensus.cc:2749] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:18.368203 23943 raft_consensus.cc:493] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.368345 23943 raft_consensus.cc:515] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.368644 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e1b4748c0244b258960b1025da47562 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.368584 23787 raft_consensus.cc:359] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.368744 23787 raft_consensus.cc:385] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.368835 23787 raft_consensus.cc:740] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.368966 23943 leader_election.cc:290] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:18.369097 23942 raft_consensus.cc:493] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.369050 23787 consensus_queue.cc:260] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.369247 23942 raft_consensus.cc:515] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.369832 23942 leader_election.cc:290] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:18.370152 23790 ts_tablet_manager.cc:1434] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251023 20:15:18.370723 23333 leader_election.cc:343] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: e44dfb4fd3d04a58b5fcc6102e055ff1
I20251023 20:15:18.370437 23790 tablet_bootstrap.cc:492] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.371887 23787 ts_tablet_manager.cc:1434] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:18.371971 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet e44dfb4fd3d04a58b5fcc6102e055ff1 with delete_type TABLET_DATA_DELETED (replaced by tablet e5368813164949ff80e27bf59dcc931a) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.372159 23787 tablet_bootstrap.cc:492] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.372220 23785 tablet_replica.cc:333] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.372391 23785 raft_consensus.cc:2243] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.372581 23785 raft_consensus.cc:2272] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.372773 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c with delete_type TABLET_DATA_DELETED (replaced by tablet 8e19f64a9ac44efba92928d0fa16819a) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:18.373060 23333 leader_election.cc:343] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e44dfb4fd3d04a58b5fcc6102e055ff1
I20251023 20:15:18.373193 23333 leader_election.cc:304] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.373340 23785 ts_tablet_manager.cc:1916] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:18.373392 23333 raft_consensus.cc:2686] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251023 20:15:18.374137 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:18.374931 23605 leader_election.cc:343] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 949e31d4a9ee4579843ed54153e22c7d
I20251023 20:15:18.375916 23787 tablet_bootstrap.cc:654] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:18.375941 23604 leader_election.cc:343] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 949e31d4a9ee4579843ed54153e22c7d
I20251023 20:15:18.376070 23604 leader_election.cc:304] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.376441 23941 raft_consensus.cc:2749] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:18.359320 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34a738510f514b088469de3d8a39ca2c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:18.378821 23536 raft_consensus.cc:2393] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:18.367471 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c with delete_type TABLET_DATA_DELETED (replaced by tablet 8e19f64a9ac44efba92928d0fa16819a) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.379237 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c7cdd3d9eaf8467ba67a5645d5e4547c" } new_table_name: "default.Table_dcdc9f5d2db74554b90c3565380b6a4b" modify_external_catalogs: true
I20251023 20:15:18.379360 23333 leader_election.cc:304] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.379930 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 1685213b588148febd6a23e1d2d6691d (table default.Table_2e1bf7e58ab84283b5f61b77865778ca [id=580a286a829e44f9b9992f3b0b7482eb]) successfully deleted
I20251023 20:15:18.379972 23943 raft_consensus.cc:3060] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:18.380944 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 83da09ee0d10435ba89a19dc378d34eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83da09ee0d10435ba89a19dc378d34eb already in progress: creating tablet
W20251023 20:15:18.381912 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 8da82e6d24b14a129b91bb5298139d07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8da82e6d24b14a129b91bb5298139d07 already in progress: creating tablet
W20251023 20:15:18.382802 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5964da70989b4e6cb77fe5d0a52ce9cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5964da70989b4e6cb77fe5d0a52ce9cf already in progress: creating tablet
I20251023 20:15:18.382841 23943 raft_consensus.cc:2749] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:18.383317 23784 tablet_replica.cc:333] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:18.383544 23784 raft_consensus.cc:2243] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.383715 23784 raft_consensus.cc:2272] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.384840 23943 raft_consensus.cc:493] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.385030 23943 raft_consensus.cc:515] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.385197 23784 ts_tablet_manager.cc:1916] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.385780 23943 leader_election.cc:290] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:18.386480 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71263f7cf0994e78b5e7baa48fd4486b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:18.386972 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71263f7cf0994e78b5e7baa48fd4486b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:18.387302 23333 leader_election.cc:343] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:18.387677 23333 leader_election.cc:343] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:18.387807 23333 leader_election.cc:304] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.388180 23943 raft_consensus.cc:2749] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:18.389257 23785 ts_tablet_manager.cc:1929] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.389387 23785 log.cc:1199] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e44dfb4fd3d04a58b5fcc6102e055ff1
I20251023 20:15:18.389412 23786 tablet_bootstrap.cc:492] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:18.389501 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 352812b56f8d4feb818a364d6adfd86d (DEFAULT_TABLE table=default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.389724 23786 ts_tablet_manager.cc:1403] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.003s
I20251023 20:15:18.389820 23785 ts_tablet_manager.cc:1950] T e44dfb4fd3d04a58b5fcc6102e055ff1 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.389937 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352812b56f8d4feb818a364d6adfd86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.390774 23786 raft_consensus.cc:359] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.391036 23786 raft_consensus.cc:385] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.391156 23786 raft_consensus.cc:740] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.391430 23786 consensus_queue.cc:260] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.391976 23786 ts_tablet_manager.cc:1434] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:18.393644 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet e44dfb4fd3d04a58b5fcc6102e055ff1 (table default.Table_15d2e91d9bbd4fa8ac2260d9359076f4 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:18.394064 23786 tablet_bootstrap.cc:492] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:18.400583 23787 tablet_bootstrap.cc:492] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.400636 23786 tablet_bootstrap.cc:654] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.400754 23787 ts_tablet_manager.cc:1403] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.003s
I20251023 20:15:18.401513 23787 raft_consensus.cc:359] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.401734 23787 raft_consensus.cc:385] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.401823 23787 raft_consensus.cc:740] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.402043 23787 consensus_queue.cc:260] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.402480 23787 ts_tablet_manager.cc:1434] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:18.402861 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.403434 23787 tablet_bootstrap.cc:492] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.405103 23787 tablet_bootstrap.cc:654] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.405128 23146 catalog_manager.cc:5649] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:18.405548 23146 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:18.409168 23785 tablet_replica.cc:333] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.410151 23784 ts_tablet_manager.cc:1929] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.410647 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.411509 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d with delete_type TABLET_DATA_DELETED (replaced by tablet 6558265ff9e249c68123577958f1401a) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.411749 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f515399c494aadbed2bd0eac171a77 with delete_type TABLET_DATA_DELETED (replaced by tablet ef1229b9e0384a82811f244d2bf4c8ce) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.411923 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.412056 23785 raft_consensus.cc:2243] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.411788 23784 log.cc:1199] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/3200dfdcc18f4a5db8ce0b2ac8ddf63c
I20251023 20:15:18.412319 23785 raft_consensus.cc:2272] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.412755 23790 tablet_bootstrap.cc:654] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.413118 23784 ts_tablet_manager.cc:1950] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:18.413239 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 352812b56f8d4feb818a364d6adfd86d (DEFAULT_TABLE table=default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.413645 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352812b56f8d4feb818a364d6adfd86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.413972 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c (table default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]) successfully deleted
I20251023 20:15:18.414378 23785 ts_tablet_manager.cc:1916] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.414660 23784 tablet_replica.cc:333] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:18.414836 23784 raft_consensus.cc:2243] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.415009 23784 raft_consensus.cc:2272] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.416326 23784 ts_tablet_manager.cc:1916] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.417999 23790 tablet_bootstrap.cc:492] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.418176 23790 ts_tablet_manager.cc:1403] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.002s
I20251023 20:15:18.419059 23790 raft_consensus.cc:359] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.419561 23790 raft_consensus.cc:385] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.419776 23790 raft_consensus.cc:740] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.420110 23790 consensus_queue.cc:260] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.421132 23790 ts_tablet_manager.cc:1434] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20251023 20:15:18.421463 23790 tablet_bootstrap.cc:492] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.422676 23785 ts_tablet_manager.cc:1929] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.422822 23785 log.cc:1199] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/3200dfdcc18f4a5db8ce0b2ac8ddf63c
I20251023 20:15:18.423200 23785 ts_tablet_manager.cc:1950] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.423282 23790 tablet_bootstrap.cc:654] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.424321 23785 tablet_replica.cc:333] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.424430 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c (table default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]) successfully deleted
I20251023 20:15:18.424512 23785 raft_consensus.cc:2243] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.424682 23785 raft_consensus.cc:2272] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.425426 23785 ts_tablet_manager.cc:1916] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.430100 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 283166bbd8004ce3bca9f2ecbdbcbf48
I20251023 20:15:18.430588 23139 catalog_manager.cc:6719] ReplaceTablet: tablet 283166bbd8004ce3bca9f2ecbdbcbf48 deleted and replaced by tablet 419e41d7a37f4b888cb242b089c28790
I20251023 20:15:18.431591 23785 ts_tablet_manager.cc:1929] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.431711 23785 log.cc:1199] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/71263f7cf0994e78b5e7baa48fd4486b
I20251023 20:15:18.432237 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.432502 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.432911 23785 ts_tablet_manager.cc:1950] T 71263f7cf0994e78b5e7baa48fd4486b P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:18.432986 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283166bbd8004ce3bca9f2ecbdbcbf48 already in progress: creating tablet
I20251023 20:15:18.433715 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 71263f7cf0994e78b5e7baa48fd4486b (table default.Table_cae543cd317a44e4816ccc5a330b51a0 [id=51c1f79e31c945309add87b2618ee778]) successfully deleted
I20251023 20:15:18.434078 23785 tablet_replica.cc:333] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.434064 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.434358 23785 raft_consensus.cc:2243] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.434563 23785 raft_consensus.cc:2272] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.435328 23785 ts_tablet_manager.cc:1916] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.436018 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c04b1c255ffa45618b6c2c9a27eec857"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.436620 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c04b1c255ffa45618b6c2c9a27eec857 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.438999 23787 tablet_bootstrap.cc:492] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.439172 23787 ts_tablet_manager.cc:1403] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.017s
I20251023 20:15:18.439972 23787 raft_consensus.cc:359] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.440172 23787 raft_consensus.cc:385] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.440269 23787 raft_consensus.cc:740] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.440338 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 352812b56f8d4feb818a364d6adfd86d (DEFAULT_TABLE table=default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.440539 23787 consensus_queue.cc:260] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.440836 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352812b56f8d4feb818a364d6adfd86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.441422 23787 ts_tablet_manager.cc:1434] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251023 20:15:18.443084 23784 ts_tablet_manager.cc:1929] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.441879 23787 tablet_bootstrap.cc:492] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.443926 23784 log.cc:1199] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d8aa8677f0ad4af6ae46b9ce731a7095
I20251023 20:15:18.444710 23785 ts_tablet_manager.cc:1929] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.444864 23785 log.cc:1199] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/2d698ffe7d154ef8b1994ffdd4b0c648
I20251023 20:15:18.445246 23784 ts_tablet_manager.cc:1950] T d8aa8677f0ad4af6ae46b9ce731a7095 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:18.445289 23785 ts_tablet_manager.cc:1950] T 2d698ffe7d154ef8b1994ffdd4b0c648 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.446184 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d8aa8677f0ad4af6ae46b9ce731a7095 (table default.Table_853f1904ea4d44919b14dc452f631d2f [id=b1abce94eb4c40b592048fca87a9e917]) successfully deleted
I20251023 20:15:18.446584 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 2d698ffe7d154ef8b1994ffdd4b0c648 (table default.Table_ea61082402e14334acc5162a368129e3 [id=1a48facca0024a5eaa670799650b9d6d]) successfully deleted
I20251023 20:15:18.447006 23785 tablet_replica.cc:333] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.447219 23785 raft_consensus.cc:2243] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.447439 23785 raft_consensus.cc:2272] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:18.447466 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283166bbd8004ce3bca9f2ecbdbcbf48 already in progress: creating tablet
I20251023 20:15:18.448271 23785 ts_tablet_manager.cc:1916] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.452334 23787 tablet_bootstrap.cc:654] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.452320 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 46f700ea3db04a4bad539c943e850a44 (DEFAULT_TABLE table=default.Table_aba84ed1ff684cc690f31c5615ec1700 [id=0b966d7b9d4d4ad1bdf1c84d63c6be5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.452734 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f700ea3db04a4bad539c943e850a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.452877 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 46f700ea3db04a4bad539c943e850a44 (DEFAULT_TABLE table=default.Table_aba84ed1ff684cc690f31c5615ec1700 [id=0b966d7b9d4d4ad1bdf1c84d63c6be5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.453269 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f700ea3db04a4bad539c943e850a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.457337 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 46f700ea3db04a4bad539c943e850a44 (DEFAULT_TABLE table=default.Table_aba84ed1ff684cc690f31c5615ec1700 [id=0b966d7b9d4d4ad1bdf1c84d63c6be5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.457820 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f700ea3db04a4bad539c943e850a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.459447 23787 tablet_bootstrap.cc:492] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.459632 23787 ts_tablet_manager.cc:1403] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20251023 20:15:18.462126 23787 raft_consensus.cc:359] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.462349 23787 raft_consensus.cc:385] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.462430 23787 raft_consensus.cc:740] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.464576 23785 ts_tablet_manager.cc:1929] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.464766 23785 log.cc:1199] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/f49a0e08de024e3b97cb2e6ac5a59c9d
I20251023 20:15:18.464661 23787 consensus_queue.cc:260] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.465284 23785 ts_tablet_manager.cc:1950] T f49a0e08de024e3b97cb2e6ac5a59c9d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.465668 23787 ts_tablet_manager.cc:1434] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:18.465981 23787 tablet_bootstrap.cc:492] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.469790 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet f49a0e08de024e3b97cb2e6ac5a59c9d (table default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]) successfully deleted
I20251023 20:15:18.470204 23785 tablet_replica.cc:333] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.470679 23785 raft_consensus.cc:2243] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.470903 23785 raft_consensus.cc:2272] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.471678 23785 ts_tablet_manager.cc:1916] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.480298 23787 tablet_bootstrap.cc:654] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.483281 23785 ts_tablet_manager.cc:1929] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.483444 23785 log.cc:1199] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/64f515399c494aadbed2bd0eac171a77
I20251023 20:15:18.483805 23785 ts_tablet_manager.cc:1950] T 64f515399c494aadbed2bd0eac171a77 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.484807 23785 tablet_replica.cc:333] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.484983 23785 raft_consensus.cc:2243] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.485188 23785 raft_consensus.cc:2272] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.485308 23369 tablet_service.cc:1505] Processing CreateTablet for tablet ac55e14745534a02a2f6a422cab6b149 (DEFAULT_TABLE table=default.Table_664e281e682d4426b64cb60b0af1ac18 [id=b08ba744dca54428bf1ef3851b865b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.485802 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac55e14745534a02a2f6a422cab6b149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.485993 23510 tablet_service.cc:1505] Processing CreateTablet for tablet ac55e14745534a02a2f6a422cab6b149 (DEFAULT_TABLE table=default.Table_664e281e682d4426b64cb60b0af1ac18 [id=b08ba744dca54428bf1ef3851b865b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.486377 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac55e14745534a02a2f6a422cab6b149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.486270 23646 tablet_service.cc:1505] Processing CreateTablet for tablet ac55e14745534a02a2f6a422cab6b149 (DEFAULT_TABLE table=default.Table_664e281e682d4426b64cb60b0af1ac18 [id=b08ba744dca54428bf1ef3851b865b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.486732 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac55e14745534a02a2f6a422cab6b149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.494060 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 64f515399c494aadbed2bd0eac171a77 (table default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]) successfully deleted
W20251023 20:15:18.495201 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe922c438a894a7aa04699b2d2897a28 consensus state has no leader
I20251023 20:15:18.497328 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.497634 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f5d04920d5d94f63a2f285d7d824d648"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.498175 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5d04920d5d94f63a2f285d7d824d648 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.496979 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ff42478856a6492aaade00ac66a02bb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.500231 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff42478856a6492aaade00ac66a02bb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.502219 23785 ts_tablet_manager.cc:1916] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:18.514919 23132 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_a216eab77a9f465982dad2f085170ddf [id=8a388a26b69d416ab99d15c024f1d3f3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2a27706ecdb4ab78bb96c10a9f6757f consensus state has no leader
I20251023 20:15:18.515200 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 0150ab3778c34c77a7e772dd6e82b32f (DEFAULT_TABLE table=default.Table_fa3c4b34d4744caf8f1fe02cf5b2326c [id=0f1dd868b5dc4f62a887203c6a06e113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.515327 23650 tablet_service.cc:1505] Processing CreateTablet for tablet a8eaa2167f82475ca2f6a110aba288ec (DEFAULT_TABLE table=default.Table_f4de4bcad2f34502a1b63889f5b5e58a [id=05efdf3ecd3d49d4bc4da65e6f8f5b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.515643 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0150ab3778c34c77a7e772dd6e82b32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.515744 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8eaa2167f82475ca2f6a110aba288ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.516698 23785 ts_tablet_manager.cc:1929] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.516835 23785 log.cc:1199] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d8aa8677f0ad4af6ae46b9ce731a7095
I20251023 20:15:18.518047 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_42ca9b5fa333476f9d6a35fcb9d0249a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:18.518395 23785 ts_tablet_manager.cc:1950] T d8aa8677f0ad4af6ae46b9ce731a7095 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:18.518595 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42ca9b5fa333476f9d6a35fcb9d0249a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.521725 23785 tablet_replica.cc:333] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.521965 23785 raft_consensus.cc:2243] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.521901 23510 tablet_service.cc:1505] Processing CreateTablet for tablet a8eaa2167f82475ca2f6a110aba288ec (DEFAULT_TABLE table=default.Table_f4de4bcad2f34502a1b63889f5b5e58a [id=05efdf3ecd3d49d4bc4da65e6f8f5b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.522133 23785 raft_consensus.cc:2272] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.523032 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8eaa2167f82475ca2f6a110aba288ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.523491 23785 ts_tablet_manager.cc:1916] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:18.526078 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b6c3e7169154e3497dd3570dca0d745 consensus state has no leader
I20251023 20:15:18.528386 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e62c8689605f4db7b6e5016d5dcda175" } modify_external_catalogs: true
I20251023 20:15:18.528607 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e62c8689605f4db7b6e5016d5dcda175" } modify_external_catalogs: true
I20251023 20:15:18.532416 23369 tablet_service.cc:1505] Processing CreateTablet for tablet a8eaa2167f82475ca2f6a110aba288ec (DEFAULT_TABLE table=default.Table_f4de4bcad2f34502a1b63889f5b5e58a [id=05efdf3ecd3d49d4bc4da65e6f8f5b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.532927 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8eaa2167f82475ca2f6a110aba288ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.533468 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 0150ab3778c34c77a7e772dd6e82b32f (DEFAULT_TABLE table=default.Table_fa3c4b34d4744caf8f1fe02cf5b2326c [id=0f1dd868b5dc4f62a887203c6a06e113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.533887 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0150ab3778c34c77a7e772dd6e82b32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.544798 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f232c9e65a8248d2b818d81486618f5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.545456 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f232c9e65a8248d2b818d81486618f5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.548197 23790 tablet_bootstrap.cc:492] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.550233 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 0150ab3778c34c77a7e772dd6e82b32f (DEFAULT_TABLE table=default.Table_fa3c4b34d4744caf8f1fe02cf5b2326c [id=0f1dd868b5dc4f62a887203c6a06e113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.550693 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0150ab3778c34c77a7e772dd6e82b32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.556146 23785 ts_tablet_manager.cc:1929] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.556337 23785 log.cc:1199] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/487d9ee765d14aa2aa5668dc48fffe20
I20251023 20:15:18.556707 23785 ts_tablet_manager.cc:1950] T 487d9ee765d14aa2aa5668dc48fffe20 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.558029 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 83dae34129794f49b43038d3b1636ea0 (DEFAULT_TABLE table=default.Table_52af8fa4ff964f16b6fe2b43e94dddd4 [id=872a29a7d5234fc98b6dd83be48499cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.558444 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83dae34129794f49b43038d3b1636ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.548377 23790 ts_tablet_manager.cc:1403] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.127s user 0.011s sys 0.013s
I20251023 20:15:18.561573 23790 raft_consensus.cc:359] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.561779 23790 raft_consensus.cc:385] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.561865 23790 raft_consensus.cc:740] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.562080 23790 consensus_queue.cc:260] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.563690 23941 consensus_queue.cc:1048] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:18.565528 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 994a555b24e4432a84601c54d422aa5d consensus state has no leader
I20251023 20:15:18.565835 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet a636a4cdeb404e9f806d8e166ea57938 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.572170 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 7e08dad1d0f643efb3a821a1b9196290
I20251023 20:15:18.574584 23941 raft_consensus.cc:493] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.574666 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 83dae34129794f49b43038d3b1636ea0 (DEFAULT_TABLE table=default.Table_52af8fa4ff964f16b6fe2b43e94dddd4 [id=872a29a7d5234fc98b6dd83be48499cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.574813 23941 raft_consensus.cc:515] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.575109 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83dae34129794f49b43038d3b1636ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.575258 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.575572 23941 leader_election.cc:290] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:18.576057 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d698ffe7d154ef8b1994ffdd4b0c648" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:18.576644 23604 leader_election.cc:343] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:18.577694 23605 leader_election.cc:343] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 2d698ffe7d154ef8b1994ffdd4b0c648
I20251023 20:15:18.577814 23605 leader_election.cc:304] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.578248 23941 raft_consensus.cc:2749] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:18.578863 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7e08dad1d0f643efb3a821a1b9196290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e08dad1d0f643efb3a821a1b9196290 already in progress: creating tablet
I20251023 20:15:18.579667 23514 tablet_service.cc:1505] Processing CreateTablet for tablet a04c65155ce048cb82b150cbe20be849 (DEFAULT_TABLE table=default.Table_d943501e515b41e6a00142359321f818 [id=e58f31a7c86c478f92c6471e1020585b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.580124 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04c65155ce048cb82b150cbe20be849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.595172 23646 tablet_service.cc:1505] Processing CreateTablet for tablet a04c65155ce048cb82b150cbe20be849 (DEFAULT_TABLE table=default.Table_d943501e515b41e6a00142359321f818 [id=e58f31a7c86c478f92c6471e1020585b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.596189 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04c65155ce048cb82b150cbe20be849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.599398 23790 ts_tablet_manager.cc:1434] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20251023 20:15:18.601186 23790 tablet_bootstrap.cc:492] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.603945 23790 tablet_bootstrap.cc:654] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.607465 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:18.608110 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7e08dad1d0f643efb3a821a1b9196290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e08dad1d0f643efb3a821a1b9196290 already in progress: creating tablet
I20251023 20:15:18.609037 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet f908f086946949829cac15418a584730
I20251023 20:15:18.613267 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:18.613967 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f908f086946949829cac15418a584730 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f908f086946949829cac15418a584730 already in progress: creating tablet
I20251023 20:15:18.614351 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:18.615006 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f908f086946949829cac15418a584730 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f908f086946949829cac15418a584730 already in progress: creating tablet
I20251023 20:15:18.620698 23942 consensus_queue.cc:1048] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:18.625805 23787 tablet_bootstrap.cc:492] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.626049 23787 ts_tablet_manager.cc:1403] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.160s user 0.039s sys 0.044s
I20251023 20:15:18.626992 23787 raft_consensus.cc:359] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.627293 23787 raft_consensus.cc:385] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.627424 23787 raft_consensus.cc:740] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.627700 23787 consensus_queue.cc:260] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.629428 23787 ts_tablet_manager.cc:1434] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:18.629530 23786 tablet_bootstrap.cc:492] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:18.629724 23786 ts_tablet_manager.cc:1403] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.236s user 0.011s sys 0.019s
I20251023 20:15:18.629881 23787 tablet_bootstrap.cc:492] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.630506 23786 raft_consensus.cc:359] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.630692 23786 raft_consensus.cc:385] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.630761 23786 raft_consensus.cc:740] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.631000 23786 consensus_queue.cc:260] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.631506 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c with delete_type TABLET_DATA_DELETED (replaced by tablet 8e19f64a9ac44efba92928d0fa16819a) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.631506 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 949e31d4a9ee4579843ed54153e22c7d with delete_type TABLET_DATA_DELETED (replaced by tablet 9c0aeac2f20c4ca48c130a0c851f290f) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.631739 23783 tablet_replica.cc:333] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:18.631934 23783 raft_consensus.cc:2243] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.632094 23783 raft_consensus.cc:2272] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.633395 23783 ts_tablet_manager.cc:1916] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.634133 23787 tablet_bootstrap.cc:654] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.637679 23786 ts_tablet_manager.cc:1434] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:18.639957 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 15ced74d0dd74e5195ed98358957d0dd
I20251023 20:15:18.642616 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet d8aa8677f0ad4af6ae46b9ce731a7095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.642876 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.643079 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.643365 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.644160 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f515399c494aadbed2bd0eac171a77 with delete_type TABLET_DATA_DELETED (replaced by tablet ef1229b9e0384a82811f244d2bf4c8ce) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.644135 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d with delete_type TABLET_DATA_DELETED (replaced by tablet 6558265ff9e249c68123577958f1401a) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.645187 23787 tablet_bootstrap.cc:492] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.645430 23787 ts_tablet_manager.cc:1403] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.006s
I20251023 20:15:18.646329 23787 raft_consensus.cc:359] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.646631 23787 raft_consensus.cc:385] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.646749 23787 raft_consensus.cc:740] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.647058 23783 ts_tablet_manager.cc:1929] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.647086 23787 consensus_queue.cc:260] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.649209 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 with delete_type TABLET_DATA_DELETED (replaced by tablet e31aebfc87734ffeb2c0b25c07cb3f70) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.649545 23787 ts_tablet_manager.cc:1434] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:18.650771 23787 tablet_bootstrap.cc:492] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.651422 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.652637 23787 tablet_bootstrap.cc:654] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.655581 23786 tablet_bootstrap.cc:492] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:18.657867 23786 tablet_bootstrap.cc:654] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.658789 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:18.659520 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 15ced74d0dd74e5195ed98358957d0dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15ced74d0dd74e5195ed98358957d0dd already in progress: creating tablet
I20251023 20:15:18.647279 23783 log.cc:1199] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/3200dfdcc18f4a5db8ce0b2ac8ddf63c
I20251023 20:15:18.660457 23783 ts_tablet_manager.cc:1950] T 3200dfdcc18f4a5db8ce0b2ac8ddf63c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:18.662137 23787 tablet_bootstrap.cc:492] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.662292 23783 tablet_replica.cc:333] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:18.662313 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 3200dfdcc18f4a5db8ce0b2ac8ddf63c (table default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]) successfully deleted
I20251023 20:15:18.662415 23787 ts_tablet_manager.cc:1403] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20251023 20:15:18.662489 23783 raft_consensus.cc:2243] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.662727 23783 raft_consensus.cc:2272] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.663281 23783 ts_tablet_manager.cc:1916] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.663237 23787 raft_consensus.cc:359] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.663516 23787 raft_consensus.cc:385] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.663653 23787 raft_consensus.cc:740] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.663990 23787 consensus_queue.cc:260] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.664889 23787 ts_tablet_manager.cc:1434] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:18.666224 23787 tablet_bootstrap.cc:492] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:18.666438 23134 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.190:34331, user_credentials={real_user=slave}} blocked reactor thread for 128729us
I20251023 20:15:18.669164 23787 tablet_bootstrap.cc:654] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.669066 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ac4b704784f2423a82ad591bbed856ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.669715 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4b704784f2423a82ad591bbed856ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.670987 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 62acc395fb294965a1ab476d37f9fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a27f192792cf41288d69ca15ab538472) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.671654 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a0edef92f99c4edebe22ae048c5dcdde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.672161 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0edef92f99c4edebe22ae048c5dcdde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.671672 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4d358925f1254129a8b42bfca11f8870" } modify_external_catalogs: true
I20251023 20:15:18.673370 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4d358925f1254129a8b42bfca11f8870" } modify_external_catalogs: true
I20251023 20:15:18.675264 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_09ad2122f20d43989c7161f291a49a4b" } new_table_name: "default.Table_c77f332f1f69484a84091dabddde5d42" modify_external_catalogs: true
I20251023 20:15:18.675346 23139 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ae910ac41fd4e349f159ab27e44375a from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.675865 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bd80c8158a014c8192a0800e8c1c32b6" } modify_external_catalogs: true
I20251023 20:15:18.676578 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bd80c8158a014c8192a0800e8c1c32b6" } modify_external_catalogs: true
I20251023 20:15:18.676616 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_888857831b2f4735a4d944a2318eae73" } new_table_name: "default.Table_567481e14a7d4ef5a57941a4edc4694c" modify_external_catalogs: true
I20251023 20:15:18.677289 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet e4bb6abe3ef848e2831839de6a28eb0f from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.677954 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet 52bc660b9da54c5d984381a267e1efa2 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.678058 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d8aa8677f0ad4af6ae46b9ce731a7095 (table default.Table_853f1904ea4d44919b14dc452f631d2f [id=b1abce94eb4c40b592048fca87a9e917]) successfully deleted
W20251023 20:15:18.678304 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283166bbd8004ce3bca9f2ecbdbcbf48 already in progress: creating tablet
I20251023 20:15:18.678715 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 487d9ee765d14aa2aa5668dc48fffe20 (table default.Table_561d6afe24b246c9909547a248a5fe7f [id=a0f9058297034e739a638f1747ddacfd]) successfully deleted
I20251023 20:15:18.679092 23787 tablet_bootstrap.cc:492] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.679337 23787 ts_tablet_manager.cc:1403] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20251023 20:15:18.680289 23787 raft_consensus.cc:359] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.680572 23787 raft_consensus.cc:385] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.680694 23787 raft_consensus.cc:740] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.681169 23787 consensus_queue.cc:260] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.682094 23787 ts_tablet_manager.cc:1434] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:18.684192 23787 tablet_bootstrap.cc:492] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.686134 23787 tablet_bootstrap.cc:654] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.679970 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 83dae34129794f49b43038d3b1636ea0 (DEFAULT_TABLE table=default.Table_52af8fa4ff964f16b6fe2b43e94dddd4 [id=872a29a7d5234fc98b6dd83be48499cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.680656 23361 tablet_service.cc:1505] Processing CreateTablet for tablet a04c65155ce048cb82b150cbe20be849 (DEFAULT_TABLE table=default.Table_d943501e515b41e6a00142359321f818 [id=e58f31a7c86c478f92c6471e1020585b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.687049 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04c65155ce048cb82b150cbe20be849. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.689330 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83dae34129794f49b43038d3b1636ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.699672 23787 tablet_bootstrap.cc:492] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.699882 23787 ts_tablet_manager.cc:1403] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251023 20:15:18.700546 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 60348d0c00c14247860b9df14ddb82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.700767 23787 raft_consensus.cc:359] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.700968 23787 raft_consensus.cc:385] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.701050 23787 raft_consensus.cc:740] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.701344 23787 consensus_queue.cc:260] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.701874 23787 ts_tablet_manager.cc:1434] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:18.702368 23942 raft_consensus.cc:493] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.702562 23942 raft_consensus.cc:515] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.703269 23942 leader_election.cc:290] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:18.703439 23942 raft_consensus.cc:493] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.703550 23942 raft_consensus.cc:515] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.703840 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42ac1c6b30e048c58cfb7fc6ad0b5e69" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:18.704092 23942 leader_election.cc:290] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:18.704787 23604 leader_election.cc:343] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:18.704939 23787 tablet_bootstrap.cc:492] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.705194 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f49a0e08de024e3b97cb2e6ac5a59c9d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:18.705816 23604 leader_election.cc:343] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:18.708887 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.709692 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42ac1c6b30e048c58cfb7fc6ad0b5e69" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:18.710036 23786 tablet_bootstrap.cc:492] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:18.710184 23786 ts_tablet_manager.cc:1403] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20251023 20:15:18.711104 23786 raft_consensus.cc:359] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.711333 23786 raft_consensus.cc:385] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.711428 23786 raft_consensus.cc:740] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.711696 23786 consensus_queue.cc:260] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.712248 23786 ts_tablet_manager.cc:1434] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251023 20:15:18.712510 23786 tablet_bootstrap.cc:492] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:18.714412 23786 tablet_bootstrap.cc:654] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.715024 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:18.721495 23605 leader_election.cc:343] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: f49a0e08de024e3b97cb2e6ac5a59c9d
I20251023 20:15:18.721680 23605 leader_election.cc:304] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.722551 24018 raft_consensus.cc:2749] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:18.722713 23605 leader_election.cc:343] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:18.722826 23605 leader_election.cc:304] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.723049 24018 raft_consensus.cc:2749] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:18.724681 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 71263f7cf0994e78b5e7baa48fd4486b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.724942 23784 tablet_replica.cc:333] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:18.725188 23784 raft_consensus.cc:2243] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.725369 23784 raft_consensus.cc:2272] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.726375 23784 ts_tablet_manager.cc:1916] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.726631 23787 tablet_bootstrap.cc:654] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.732854 23784 ts_tablet_manager.cc:1929] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.733131 23784 log.cc:1199] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/71263f7cf0994e78b5e7baa48fd4486b
I20251023 20:15:18.733538 23784 ts_tablet_manager.cc:1950] T 71263f7cf0994e78b5e7baa48fd4486b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:18.734138 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 71263f7cf0994e78b5e7baa48fd4486b (table default.Table_cae543cd317a44e4816ccc5a330b51a0 [id=51c1f79e31c945309add87b2618ee778]) successfully deleted
I20251023 20:15:18.734549 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet f49a0e08de024e3b97cb2e6ac5a59c9d with delete_type TABLET_DATA_DELETED (replaced by tablet 6558265ff9e249c68123577958f1401a) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.734861 23784 tablet_replica.cc:333] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:18.735075 23784 raft_consensus.cc:2243] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.735299 23784 raft_consensus.cc:2272] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.736064 23784 ts_tablet_manager.cc:1916] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.741513 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a636a4cdeb404e9f806d8e166ea57938
I20251023 20:15:18.742070 23142 catalog_manager.cc:6719] ReplaceTablet: tablet a636a4cdeb404e9f806d8e166ea57938 deleted and replaced by tablet eb5a6d9a4509431f97b69a3b138b5829
I20251023 20:15:18.742550 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.742743 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d0e61de7a93d450d9ed79eef82d7e5e3
W20251023 20:15:18.743490 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 040a44e272b7410c98c109fbceb19192 consensus state has no leader
W20251023 20:15:18.743881 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f47ddde9f3d4042bc84853332e21950 consensus state has no leader
I20251023 20:15:18.745369 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.745493 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.745672 24018 raft_consensus.cc:493] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.745891 24018 raft_consensus.cc:515] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.746726 24018 leader_election.cc:290] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:18.747438 23605 leader_election.cc:343] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 64f515399c494aadbed2bd0eac171a77
I20251023 20:15:18.747634 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64f515399c494aadbed2bd0eac171a77" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:18.748114 23604 leader_election.cc:343] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:18.748247 23604 leader_election.cc:304] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.748522 24018 raft_consensus.cc:2749] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:18.750057 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.750279 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.758479 23787 tablet_bootstrap.cc:492] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.758699 23787 ts_tablet_manager.cc:1403] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.003s
I20251023 20:15:18.759004 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.759613 23787 raft_consensus.cc:359] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.759922 23787 raft_consensus.cc:385] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.759995 23787 raft_consensus.cc:740] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.760238 23787 consensus_queue.cc:260] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.762727 23787 ts_tablet_manager.cc:1434] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:18.765113 23787 tablet_bootstrap.cc:492] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.766237 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.766983 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 6a54470d088748b0be22ab9db5a25f87 (DEFAULT_TABLE table=default.Table_b93a09969d41431aa77623ab62d6f6e7 [id=a48d0e1e007c40d59766c7d0ef778956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.767249 23787 tablet_bootstrap.cc:654] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.767472 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 6a54470d088748b0be22ab9db5a25f87 (DEFAULT_TABLE table=default.Table_b93a09969d41431aa77623ab62d6f6e7 [id=a48d0e1e007c40d59766c7d0ef778956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.767957 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a54470d088748b0be22ab9db5a25f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.768074 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a54470d088748b0be22ab9db5a25f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.771107 24018 raft_consensus.cc:493] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:18.771225 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a27f192792cf41288d69ca15ab538472
I20251023 20:15:18.771237 24018 raft_consensus.cc:515] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.771785 24018 leader_election.cc:290] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:18.771860 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:18.772375 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.772867 23139 catalog_manager.cc:6719] ReplaceTablet: tablet 1ae910ac41fd4e349f159ab27e44375a deleted and replaced by tablet 72408899f710470baaf6fdb268cb04c8
I20251023 20:15:18.773301 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.774034 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:18.774497 23605 leader_election.cc:343] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: d8aa8677f0ad4af6ae46b9ce731a7095
I20251023 20:15:18.774516 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:18.775544 23604 leader_election.cc:343] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: d8aa8677f0ad4af6ae46b9ce731a7095
I20251023 20:15:18.775684 23604 leader_election.cc:304] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:18.775918 24018 raft_consensus.cc:2749] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:18.781386 23787 tablet_bootstrap.cc:492] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.781704 23787 ts_tablet_manager.cc:1403] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20251023 20:15:18.782804 23787 raft_consensus.cc:359] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.782956 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 6a54470d088748b0be22ab9db5a25f87 (DEFAULT_TABLE table=default.Table_b93a09969d41431aa77623ab62d6f6e7 [id=a48d0e1e007c40d59766c7d0ef778956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.783092 23787 raft_consensus.cc:385] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.783179 23787 raft_consensus.cc:740] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.783412 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a54470d088748b0be22ab9db5a25f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.783396 23787 consensus_queue.cc:260] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.785219 23787 ts_tablet_manager.cc:1434] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:18.785240 23376 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.785542 23787 tablet_bootstrap.cc:492] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.785516 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.787315 23787 tablet_bootstrap.cc:654] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.789383 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 419e41d7a37f4b888cb242b089c28790 (DEFAULT_TABLE table=default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.789803 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419e41d7a37f4b888cb242b089c28790. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.793404 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 165d62dd42da460faccec706f08272a1 (DEFAULT_TABLE table=default.Table_ce123b77e83b41c4bf1fc149512c47d7 [id=d3624aa88af54933bf909725fd573d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.793809 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165d62dd42da460faccec706f08272a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.794024 23790 tablet_bootstrap.cc:492] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.794193 23790 ts_tablet_manager.cc:1403] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.193s user 0.002s sys 0.001s
I20251023 20:15:18.795022 23790 raft_consensus.cc:359] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.795217 23790 raft_consensus.cc:385] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.795276 23790 raft_consensus.cc:740] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.795548 23790 consensus_queue.cc:260] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.797437 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 (DEFAULT_TABLE table=default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.797837 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ae6a6e58274b8baf952eaaefde7127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.798718 23790 ts_tablet_manager.cc:1434] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:18.801138 23790 tablet_bootstrap.cc:492] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.801393 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 25079532547d4175802b84877790662d (DEFAULT_TABLE table=default.Table_a21f26a0638f42d682abb0448b2a53e5 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.801750 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25079532547d4175802b84877790662d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.802307 23787 tablet_bootstrap.cc:492] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.802485 23787 ts_tablet_manager.cc:1403] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20251023 20:15:18.803370 23790 tablet_bootstrap.cc:654] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.803306 23787 raft_consensus.cc:359] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.803596 23787 raft_consensus.cc:385] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.803681 23787 raft_consensus.cc:740] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.803927 23787 consensus_queue.cc:260] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:18.804425 23134 catalog_manager.cc:5101] could not convert tablet's 34a738510f514b088469de3d8a39ca2c schema from PB: OK
I20251023 20:15:18.804569 23787 ts_tablet_manager.cc:1434] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:18.804888 23787 tablet_bootstrap.cc:492] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:18.805239 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 7e08dad1d0f643efb3a821a1b9196290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e08dad1d0f643efb3a821a1b9196290 already in progress: creating tablet
W20251023 20:15:18.805466 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet f908f086946949829cac15418a584730 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f908f086946949829cac15418a584730 already in progress: creating tablet
W20251023 20:15:18.805660 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 15ced74d0dd74e5195ed98358957d0dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15ced74d0dd74e5195ed98358957d0dd already in progress: creating tablet
I20251023 20:15:18.806149 23501 tablet_service.cc:1505] Processing CreateTablet for tablet a86a25659ec44d7c825783036342ed45 (DEFAULT_TABLE table=default.Table_9899db7c169244f38fdb46b1fd263e1a [id=7c367be5410c4fae9bff6652c2f15532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.806151 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_15d2e91d9bbd4fa8ac2260d9359076f4" } new_table_name: "default.Table_7c7ed77bec2746e0ba85dd8598b4c364" modify_external_catalogs: true
I20251023 20:15:18.806525 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86a25659ec44d7c825783036342ed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.806702 23787 tablet_bootstrap.cc:654] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.809242 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8ff1c10b4f2f49dd950540c129481320"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.809780 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ff1c10b4f2f49dd950540c129481320 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.811214 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_baf578c1b5e748e09f244f10685992a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.811718 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baf578c1b5e748e09f244f10685992a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.812191 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet 4b6c3e7169154e3497dd3570dca0d745 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:18.814765 23650 tablet_service.cc:1505] Processing CreateTablet for tablet a86a25659ec44d7c825783036342ed45 (DEFAULT_TABLE table=default.Table_9899db7c169244f38fdb46b1fd263e1a [id=7c367be5410c4fae9bff6652c2f15532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.815176 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86a25659ec44d7c825783036342ed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.816433 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 419e41d7a37f4b888cb242b089c28790 (DEFAULT_TABLE table=default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.816831 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419e41d7a37f4b888cb242b089c28790. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.818886 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 165d62dd42da460faccec706f08272a1 (DEFAULT_TABLE table=default.Table_ce123b77e83b41c4bf1fc149512c47d7 [id=d3624aa88af54933bf909725fd573d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.819244 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165d62dd42da460faccec706f08272a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.823971 23646 tablet_service.cc:1505] Processing CreateTablet for tablet b5cb6b4e2e5e4a2f9183111cfd54ec90 (DEFAULT_TABLE table=default.Table_57771b43db0747d68d0dea8e558351e0 [id=c35038526af34fc682bc386c5c80cd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.824447 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5cb6b4e2e5e4a2f9183111cfd54ec90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.828012 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.828402 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f with delete_type TABLET_DATA_DELETED (replaced by tablet 8289ce37fa074891beb584bf2fff9747) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.831226 23784 ts_tablet_manager.cc:1929] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.831367 23784 log.cc:1199] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/f49a0e08de024e3b97cb2e6ac5a59c9d
I20251023 20:15:18.831712 23784 ts_tablet_manager.cc:1950] T f49a0e08de024e3b97cb2e6ac5a59c9d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:18.832688 23536 raft_consensus.cc:1275] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Refusing update from remote peer 9bc6522db11e4960a9aad84435371a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:18.833101 23787 tablet_bootstrap.cc:492] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.833259 23787 ts_tablet_manager.cc:1403] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20251023 20:15:18.833312 23941 consensus_queue.cc:1048] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:18.834127 23787 raft_consensus.cc:359] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.834311 23787 raft_consensus.cc:385] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.834388 23787 raft_consensus.cc:740] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.834574 23787 consensus_queue.cc:260] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.835227 23787 ts_tablet_manager.cc:1434] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:18.835798 23787 tablet_bootstrap.cc:492] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.838162 23787 tablet_bootstrap.cc:654] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.837186 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 (DEFAULT_TABLE table=default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.839233 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ae6a6e58274b8baf952eaaefde7127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.843780 23399 raft_consensus.cc:1275] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Refusing update from remote peer 9bc6522db11e4960a9aad84435371a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:18.844210 23787 tablet_bootstrap.cc:492] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.844576 23941 consensus_queue.cc:1048] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:18.844717 23787 ts_tablet_manager.cc:1403] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20251023 20:15:18.846005 23787 raft_consensus.cc:359] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.846311 23787 raft_consensus.cc:385] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.846424 23787 raft_consensus.cc:740] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.846733 23787 consensus_queue.cc:260] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.847591 23787 ts_tablet_manager.cc:1434] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:18.847965 23787 tablet_bootstrap.cc:492] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.849201 23783 ts_tablet_manager.cc:1929] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.849438 23783 log.cc:1199] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/949e31d4a9ee4579843ed54153e22c7d
I20251023 20:15:18.849999 23783 ts_tablet_manager.cc:1950] T 949e31d4a9ee4579843ed54153e22c7d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:18.851504 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b380171aa08d439eadeddb7b4c6e6a6a" } new_table_name: "default.Table_09442523fb5d4e9d856cdcfba51fb911" modify_external_catalogs: true
I20251023 20:15:18.853004 23143 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:18.854806 23362 tablet_service.cc:1505] Processing CreateTablet for tablet b5cb6b4e2e5e4a2f9183111cfd54ec90 (DEFAULT_TABLE table=default.Table_57771b43db0747d68d0dea8e558351e0 [id=c35038526af34fc682bc386c5c80cd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.855233 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5cb6b4e2e5e4a2f9183111cfd54ec90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.856247 23787 tablet_bootstrap.cc:654] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.857231 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.859032 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 949e31d4a9ee4579843ed54153e22c7d (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]) successfully deleted
I20251023 20:15:18.862046 23362 tablet_service.cc:1505] Processing CreateTablet for tablet a86a25659ec44d7c825783036342ed45 (DEFAULT_TABLE table=default.Table_9899db7c169244f38fdb46b1fd263e1a [id=7c367be5410c4fae9bff6652c2f15532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.862648 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a86a25659ec44d7c825783036342ed45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.863644 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:18.865633 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5368813164949ff80e27bf59dcc931a consensus state has no leader
I20251023 20:15:18.870388 23787 tablet_bootstrap.cc:492] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.870698 23787 ts_tablet_manager.cc:1403] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20251023 20:15:18.871404 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 4b6c3e7169154e3497dd3570dca0d745
I20251023 20:15:18.871644 23787 raft_consensus.cc:359] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.871856 23787 raft_consensus.cc:385] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.871886 23141 catalog_manager.cc:6719] ReplaceTablet: tablet 4b6c3e7169154e3497dd3570dca0d745 deleted and replaced by tablet 19c58a406d564b048fb16a85aa0b754b
I20251023 20:15:18.871935 23787 raft_consensus.cc:740] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.872208 23787 consensus_queue.cc:260] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.872411 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.872699 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.873042 23787 ts_tablet_manager.cc:1434] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20251023 20:15:18.874899 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9c5b69df4c04365b1ac1457fe8300cc consensus state has no leader
I20251023 20:15:18.876420 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 25079532547d4175802b84877790662d (DEFAULT_TABLE table=default.Table_a21f26a0638f42d682abb0448b2a53e5 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.878404 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 419e41d7a37f4b888cb242b089c28790 (DEFAULT_TABLE table=default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.879001 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 419e41d7a37f4b888cb242b089c28790. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.881644 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 (DEFAULT_TABLE table=default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:18.882037 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a27f192792cf41288d69ca15ab538472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a27f192792cf41288d69ca15ab538472 already in progress: creating tablet
I20251023 20:15:18.882223 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 00ec606cbf834ccd8762e034b520e58e (DEFAULT_TABLE table=default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:18.882277 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1ae910ac41fd4e349f159ab27e44375a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ae910ac41fd4e349f159ab27e44375a already in progress: creating tablet
W20251023 20:15:18.882485 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d0e61de7a93d450d9ed79eef82d7e5e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0e61de7a93d450d9ed79eef82d7e5e3 already in progress: creating tablet
I20251023 20:15:18.882675 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ae6a6e58274b8baf952eaaefde7127. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:18.882901 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a636a4cdeb404e9f806d8e166ea57938 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a636a4cdeb404e9f806d8e166ea57938 already in progress: creating tablet
I20251023 20:15:18.883718 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 60348d0c00c14247860b9df14ddb82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.884027 23785 tablet_replica.cc:333] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:18.884215 23785 raft_consensus.cc:2243] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.884402 23785 raft_consensus.cc:2272] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.885597 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ec606cbf834ccd8762e034b520e58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.889608 23785 ts_tablet_manager.cc:1916] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.889825 23783 tablet_replica.cc:333] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:18.890044 23783 raft_consensus.cc:2243] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.890223 23783 raft_consensus.cc:2272] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.890457 23790 tablet_bootstrap.cc:492] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.890714 23790 ts_tablet_manager.cc:1403] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.090s user 0.000s sys 0.004s
I20251023 20:15:18.891609 23790 raft_consensus.cc:359] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.891793 23790 raft_consensus.cc:385] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.891880 23790 raft_consensus.cc:740] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.892170 23790 consensus_queue.cc:260] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.892649 23783 ts_tablet_manager.cc:1916] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.893392 23790 ts_tablet_manager.cc:1434] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:18.893743 23790 tablet_bootstrap.cc:492] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.895848 23790 tablet_bootstrap.cc:654] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.905436 23783 ts_tablet_manager.cc:1929] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.906217 23786 tablet_bootstrap.cc:492] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:18.906392 23786 ts_tablet_manager.cc:1403] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.194s user 0.003s sys 0.000s
I20251023 20:15:18.906528 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 25079532547d4175802b84877790662d (DEFAULT_TABLE table=default.Table_a21f26a0638f42d682abb0448b2a53e5 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.906921 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25079532547d4175802b84877790662d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.907240 23786 raft_consensus.cc:359] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.907433 23786 raft_consensus.cc:385] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.907497 23786 raft_consensus.cc:740] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.907819 23786 consensus_queue.cc:260] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.908591 23783 log.cc:1199] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d8aa8677f0ad4af6ae46b9ce731a7095
I20251023 20:15:18.909204 23783 ts_tablet_manager.cc:1950] T d8aa8677f0ad4af6ae46b9ce731a7095 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:18.910337 23786 ts_tablet_manager.cc:1434] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251023 20:15:18.912448 24005 mvcc.cc:204] Tried to move back new op lower bound from 7214082125130768384 to 7214082121531166720. Current Snapshot: MvccSnapshot[applied={T|T < 7214082125130768384}]
I20251023 20:15:18.914098 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 00ec606cbf834ccd8762e034b520e58e (DEFAULT_TABLE table=default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.914776 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet c95db7b5b9fd46acae8ffa7119915907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.916000 23144 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:18.917851 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 165d62dd42da460faccec706f08272a1 (DEFAULT_TABLE table=default.Table_ce123b77e83b41c4bf1fc149512c47d7 [id=d3624aa88af54933bf909725fd573d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.919587 23786 tablet_bootstrap.cc:492] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:18.919893 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 165d62dd42da460faccec706f08272a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.924072 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet f49a0e08de024e3b97cb2e6ac5a59c9d (table default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]) successfully deleted
W20251023 20:15:18.924314 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d0e61de7a93d450d9ed79eef82d7e5e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0e61de7a93d450d9ed79eef82d7e5e3 already in progress: creating tablet
W20251023 20:15:18.924549 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a636a4cdeb404e9f806d8e166ea57938 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a636a4cdeb404e9f806d8e166ea57938 already in progress: creating tablet
W20251023 20:15:18.924737 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a27f192792cf41288d69ca15ab538472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a27f192792cf41288d69ca15ab538472 already in progress: creating tablet
W20251023 20:15:18.924930 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1ae910ac41fd4e349f159ab27e44375a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ae910ac41fd4e349f159ab27e44375a already in progress: creating tablet
I20251023 20:15:18.924937 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:18.925163 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 4b6c3e7169154e3497dd3570dca0d745 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b6c3e7169154e3497dd3570dca0d745 already in progress: creating tablet
I20251023 20:15:18.925768 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 64f515399c494aadbed2bd0eac171a77 with delete_type TABLET_DATA_DELETED (replaced by tablet ef1229b9e0384a82811f244d2bf4c8ce) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.926800 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 00ec606cbf834ccd8762e034b520e58e (DEFAULT_TABLE table=default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.927211 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ec606cbf834ccd8762e034b520e58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.929272 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d39a64892c5b49d99dccfbe4163d6f53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.929951 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d39a64892c5b49d99dccfbe4163d6f53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.934139 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_465a2fe41b184ff49857af876a59642b" } new_table_name: "default.Table_39094702f1ba4174a5eb05a40213e745" modify_external_catalogs: true
I20251023 20:15:18.934499 23785 ts_tablet_manager.cc:1929] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.934630 23785 log.cc:1199] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/60348d0c00c14247860b9df14ddb82b9
I20251023 20:15:18.935003 23785 ts_tablet_manager.cc:1950] T 60348d0c00c14247860b9df14ddb82b9 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:18.935151 24008 mvcc.cc:204] Tried to move back new op lower bound from 7214082125130768384 to 7214082121531166720. Current Snapshot: MvccSnapshot[applied={T|T < 7214082125130768384}]
I20251023 20:15:18.935704 23787 tablet_bootstrap.cc:492] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.937896 23787 tablet_bootstrap.cc:654] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.940730 23783 tablet_replica.cc:333] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:18.940930 23783 raft_consensus.cc:2243] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.941238 23783 raft_consensus.cc:2272] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.941466 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d8aa8677f0ad4af6ae46b9ce731a7095 (table default.Table_853f1904ea4d44919b14dc452f631d2f [id=b1abce94eb4c40b592048fca87a9e917]) successfully deleted
I20251023 20:15:18.941764 23783 ts_tablet_manager.cc:1916] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:18.943390 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7301d6121c9d4c8dbc642e895aa4e9f6 consensus state has no leader
I20251023 20:15:18.945255 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ec606cbf834ccd8762e034b520e58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.946288 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d880d87e1b2244b8ba05b6ff6653e35b" } modify_external_catalogs: true
I20251023 20:15:18.946468 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d880d87e1b2244b8ba05b6ff6653e35b" } modify_external_catalogs: true
I20251023 20:15:18.947733 24036 tablet.cc:1722] T 34a738510f514b088469de3d8a39ca2c P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:18.953562 23146 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:18.953634 23499 tablet_service.cc:1505] Processing CreateTablet for tablet b5cb6b4e2e5e4a2f9183111cfd54ec90 (DEFAULT_TABLE table=default.Table_57771b43db0747d68d0dea8e558351e0 [id=c35038526af34fc682bc386c5c80cd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:18.954197 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5cb6b4e2e5e4a2f9183111cfd54ec90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.958343 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:18.958925 24035 tablet.cc:1722] T 34a738510f514b088469de3d8a39ca2c P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:18.960036 23783 ts_tablet_manager.cc:1929] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.960234 23783 log.cc:1199] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/71263f7cf0994e78b5e7baa48fd4486b
I20251023 20:15:18.961314 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25079532547d4175802b84877790662d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:18.963261 23784 tablet_replica.cc:333] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:18.964422 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:18.961795 23783 ts_tablet_manager.cc:1950] T 71263f7cf0994e78b5e7baa48fd4486b P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:18.965313 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:18.966212 23787 tablet_bootstrap.cc:492] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:18.966384 23787 ts_tablet_manager.cc:1403] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.002s
I20251023 20:15:18.967180 23787 raft_consensus.cc:359] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.967386 23787 raft_consensus.cc:385] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.967445 23787 raft_consensus.cc:740] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.967680 23787 consensus_queue.cc:260] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:18.968154 23787 ts_tablet_manager.cc:1434] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:18.968449 23787 tablet_bootstrap.cc:492] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:18.964505 23784 raft_consensus.cc:2243] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.970059 23784 raft_consensus.cc:2272] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.970217 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.972719 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_52af8fa4ff964f16b6fe2b43e94dddd4" } modify_external_catalogs: true
I20251023 20:15:18.973693 23787 tablet_bootstrap.cc:654] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.973485 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6de6ff0130d74ad894cfa487113e1b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.974427 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6de6ff0130d74ad894cfa487113e1b19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:18.975099 23535 raft_consensus.cc:1275] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Refusing update from remote peer 9bc6522db11e4960a9aad84435371a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:18.975760 23941 consensus_queue.cc:1048] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:18.977820 23783 tablet_replica.cc:333] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:18.977802 23399 raft_consensus.cc:1275] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Refusing update from remote peer 9bc6522db11e4960a9aad84435371a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:18.978036 23783 raft_consensus.cc:2243] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.978168 23783 raft_consensus.cc:2272] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.978649 23783 ts_tablet_manager.cc:1916] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.986630 23790 tablet_bootstrap.cc:492] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:18.987071 23790 ts_tablet_manager.cc:1403] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.093s user 0.019s sys 0.026s
I20251023 20:15:18.988405 23790 raft_consensus.cc:359] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.988775 23784 ts_tablet_manager.cc:1916] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.989630 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 71263f7cf0994e78b5e7baa48fd4486b (table default.Table_cae543cd317a44e4816ccc5a330b51a0 [id=51c1f79e31c945309add87b2618ee778]) successfully deleted
I20251023 20:15:18.990069 23790 raft_consensus.cc:385] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:18.990391 23941 consensus_queue.cc:1048] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:18.990769 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:18.991221 23790 raft_consensus.cc:740] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:18.991456 23783 ts_tablet_manager.cc:1929] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:18.991631 23783 log.cc:1199] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/2d698ffe7d154ef8b1994ffdd4b0c648
I20251023 20:15:18.992040 23783 ts_tablet_manager.cc:1950] T 2d698ffe7d154ef8b1994ffdd4b0c648 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:18.991816 23790 consensus_queue.cc:260] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:18.993281 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 2d698ffe7d154ef8b1994ffdd4b0c648 (table default.Table_ea61082402e14334acc5162a368129e3 [id=1a48facca0024a5eaa670799650b9d6d]) successfully deleted
I20251023 20:15:18.993398 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_52af8fa4ff964f16b6fe2b43e94dddd4" } modify_external_catalogs: true
I20251023 20:15:18.993695 23790 ts_tablet_manager.cc:1434] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:18.994330 23783 tablet_replica.cc:333] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:18.994551 23783 raft_consensus.cc:2243] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:18.994771 23783 raft_consensus.cc:2272] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:18.995357 23790 tablet_bootstrap.cc:492] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:18.995589 23783 ts_tablet_manager.cc:1916] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:18.927232 23786 tablet_bootstrap.cc:654] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:18.998996 23790 tablet_bootstrap.cc:654] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:18.993896 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 15ced74d0dd74e5195ed98358957d0dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15ced74d0dd74e5195ed98358957d0dd already in progress: creating tablet
I20251023 20:15:18.996569 23146 catalog_manager.cc:6194] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3] alter complete (version 2)
I20251023 20:15:18.998278 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_49f2383c82e44457827030102d18d5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:18.999822 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49f2383c82e44457827030102d18d5bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.000000 23147 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:19.001992 23787 tablet_bootstrap.cc:492] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.002199 23787 ts_tablet_manager.cc:1403] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.009s
I20251023 20:15:19.002970 23787 raft_consensus.cc:359] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.003201 23787 raft_consensus.cc:385] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.003331 23787 raft_consensus.cc:740] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.003614 23787 consensus_queue.cc:260] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.005741 23787 ts_tablet_manager.cc:1434] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251023 20:15:19.006198 23787 tablet_bootstrap.cc:492] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.008311 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 with delete_type TABLET_DATA_DELETED (replaced by tablet edec01a63a3046d9ae5a372fc9068b52) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.009680 23787 tablet_bootstrap.cc:654] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.009799 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 60348d0c00c14247860b9df14ddb82b9 (table default.Table_baf231bb8eaa40eeb01169ddcf5161bd [id=94828b76eb8a4afdac4136fc626b6147]) successfully deleted
W20251023 20:15:19.010035 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 4b6c3e7169154e3497dd3570dca0d745 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b6c3e7169154e3497dd3570dca0d745 already in progress: creating tablet
W20251023 20:15:19.010274 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb915d114fc0421999d079c23cad8eb5 already in progress: creating tablet
I20251023 20:15:19.011970 23790 tablet_bootstrap.cc:492] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:19.012154 23790 ts_tablet_manager.cc:1403] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251023 20:15:19.012573 23783 ts_tablet_manager.cc:1929] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.013527 23783 log.cc:1199] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/64f515399c494aadbed2bd0eac171a77
I20251023 20:15:19.013716 23790 raft_consensus.cc:359] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.013942 23790 raft_consensus.cc:385] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.014055 23783 ts_tablet_manager.cc:1950] T 64f515399c494aadbed2bd0eac171a77 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.014061 23790 raft_consensus.cc:740] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.014125 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6d625b139835401587825737c53000e3" } modify_external_catalogs: true
I20251023 20:15:19.014395 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6d625b139835401587825737c53000e3" } modify_external_catalogs: true
I20251023 20:15:19.014377 23790 consensus_queue.cc:260] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.015267 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7c5125a919774476a6ec9f806c2d329e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.015573 23790 ts_tablet_manager.cc:1434] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:19.015915 23790 tablet_bootstrap.cc:492] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:19.016237 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c5125a919774476a6ec9f806c2d329e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.015667 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_00e41d94627b471b9753905690b03a49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.016506 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 64f515399c494aadbed2bd0eac171a77 (table default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]) successfully deleted
W20251023 20:15:19.016726 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00e41d94627b471b9753905690b03a49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.016808 23783 tablet_replica.cc:333] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.017030 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8ff1c10b4f2f49dd950540c129481320" } modify_external_catalogs: true
I20251023 20:15:19.017189 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8ff1c10b4f2f49dd950540c129481320" } modify_external_catalogs: true
I20251023 20:15:19.017050 23783 raft_consensus.cc:2243] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.017627 23790 tablet_bootstrap.cc:654] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.017671 23783 raft_consensus.cc:2272] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.017887 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_20ccdc7f398e41459cbbb75ffa904e18" } new_table_name: "default.Table_6d990f17621b44f78107cef2a0994e8e" modify_external_catalogs: true
I20251023 20:15:19.018471 23783 ts_tablet_manager.cc:1916] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.019768 23787 tablet_bootstrap.cc:492] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.019979 23787 ts_tablet_manager.cc:1403] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251023 20:15:19.021279 23787 raft_consensus.cc:359] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.021584 23787 raft_consensus.cc:385] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.021689 23787 raft_consensus.cc:740] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.022054 23787 consensus_queue.cc:260] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.024073 23787 ts_tablet_manager.cc:1434] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:19.025852 23787 tablet_bootstrap.cc:492] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.026818 23941 raft_consensus.cc:493] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.027025 23941 raft_consensus.cc:515] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.027621 23941 leader_election.cc:290] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:19.028399 23605 leader_election.cc:343] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 60348d0c00c14247860b9df14ddb82b9
I20251023 20:15:19.029206 23787 tablet_bootstrap.cc:654] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:19.030330 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af899c01b4af46d39d27f984cc3d5940 consensus state has no leader
I20251023 20:15:19.030915 23790 tablet_bootstrap.cc:492] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:19.031139 23790 ts_tablet_manager.cc:1403] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251023 20:15:19.031970 23790 raft_consensus.cc:359] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.032316 23790 raft_consensus.cc:385] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.032434 23790 raft_consensus.cc:740] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.032747 23790 consensus_queue.cc:260] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.033010 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_391ecae9d82d4a9089bc2d004ca92bc6" } new_table_name: "default.Table_01e28a3f1691446384d30911e4894273" modify_external_catalogs: true
I20251023 20:15:19.034152 23790 ts_tablet_manager.cc:1434] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:19.035087 23790 tablet_bootstrap.cc:492] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:19.037317 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a87b53598d1a4c0b9324ab792449a67d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.038188 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a87b53598d1a4c0b9324ab792449a67d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.038451 23783 ts_tablet_manager.cc:1929] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:19.038707 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e76f2e26dcd4cdfbf72de899f5a1458 consensus state has no leader
I20251023 20:15:19.038832 24035 tablet.cc:1722] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:19.039310 23783 log.cc:1199] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/f49a0e08de024e3b97cb2e6ac5a59c9d
I20251023 20:15:19.039467 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_79be15cb4350452b90277004ec803c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.040004 23783 ts_tablet_manager.cc:1950] T f49a0e08de024e3b97cb2e6ac5a59c9d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.043087 23142 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998]). Expected version 4 got 0
I20251023 20:15:19.043522 23783 tablet_replica.cc:333] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.043704 23783 raft_consensus.cc:2243] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.044095 23783 raft_consensus.cc:2272] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.045109 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet f49a0e08de024e3b97cb2e6ac5a59c9d (table default.Table_a6d0f3fe1c734fac92db5f6868a805de [id=991a6048a0894f9f95f52a8f0b881db6]) successfully deleted
W20251023 20:15:19.045361 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79be15cb4350452b90277004ec803c8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.046379 23783 ts_tablet_manager.cc:1916] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.047250 23790 tablet_bootstrap.cc:654] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.048190 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_aab23aa8d50a4902b09495b3fa1ea6ca" } modify_external_catalogs: true
I20251023 20:15:19.048408 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_aab23aa8d50a4902b09495b3fa1ea6ca" } modify_external_catalogs: true
I20251023 20:15:19.048574 24036 tablet.cc:1722] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:19.049274 23787 tablet_bootstrap.cc:492] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.049459 23787 ts_tablet_manager.cc:1403] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20251023 20:15:19.050225 23787 raft_consensus.cc:359] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.050470 23787 raft_consensus.cc:385] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.051038 23787 raft_consensus.cc:740] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.051357 23787 consensus_queue.cc:260] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.051451 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_fab9daaa8e644d2a9681e2229696c2bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.051990 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fab9daaa8e644d2a9681e2229696c2bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.052608 23787 ts_tablet_manager.cc:1434] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:19.053563 23787 tablet_bootstrap.cc:492] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.055626 23787 tablet_bootstrap.cc:654] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.055649 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 62acc395fb294965a1ab476d37f9fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a27f192792cf41288d69ca15ab538472) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.055903 23785 tablet_replica.cc:333] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:19.056107 23785 raft_consensus.cc:2243] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.056267 23785 raft_consensus.cc:2272] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.057124 23785 ts_tablet_manager.cc:1916] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.061256 23783 ts_tablet_manager.cc:1929] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.061337 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.061582 23783 log.cc:1199] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/02b79ba69a1c4f8ebc7ecb795ac0d1d5
I20251023 20:15:19.062179 23783 ts_tablet_manager.cc:1950] T 02b79ba69a1c4f8ebc7ecb795ac0d1d5 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.063136 23787 tablet_bootstrap.cc:492] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.063519 23787 ts_tablet_manager.cc:1403] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20251023 20:15:19.063704 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 02b79ba69a1c4f8ebc7ecb795ac0d1d5 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
I20251023 20:15:19.063535 23783 tablet_replica.cc:333] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.064569 23787 raft_consensus.cc:359] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.066376 23790 tablet_bootstrap.cc:492] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:19.066592 23790 ts_tablet_manager.cc:1403] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20251023 20:15:19.066710 23785 ts_tablet_manager.cc:1929] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.067234 23785 log.cc:1199] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/62acc395fb294965a1ab476d37f9fab1
I20251023 20:15:19.064801 23783 raft_consensus.cc:2243] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.067711 23787 raft_consensus.cc:385] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.067827 23783 raft_consensus.cc:2272] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.067843 23787 raft_consensus.cc:740] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.069213 23787 consensus_queue.cc:260] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.069931 23783 ts_tablet_manager.cc:1916] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.070881 23785 ts_tablet_manager.cc:1950] T 62acc395fb294965a1ab476d37f9fab1 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:19.071779 23785 tablet_replica.cc:333] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:19.071971 23785 raft_consensus.cc:2243] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.072154 23785 raft_consensus.cc:2272] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.067454 23790 raft_consensus.cc:359] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.072386 23790 raft_consensus.cc:385] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.072491 23790 raft_consensus.cc:740] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.072990 23143 catalog_manager.cc:6194] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998] alter complete (version 4)
I20251023 20:15:19.073318 23790 consensus_queue.cc:260] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.073542 23787 ts_tablet_manager.cc:1434] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251023 20:15:19.074034 23787 tablet_bootstrap.cc:492] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.075116 23785 ts_tablet_manager.cc:1916] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.076304 23790 ts_tablet_manager.cc:1434] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:19.077052 23790 tablet_bootstrap.cc:492] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:19.078462 24018 raft_consensus.cc:493] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.078614 24018 raft_consensus.cc:515] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.079226 24018 leader_election.cc:290] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.079751 23790 tablet_bootstrap.cc:654] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.083632 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bcdec7c16c943079e20bb486aecb431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.083954 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c95db7b5b9fd46acae8ffa7119915907" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.084443 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 6558265ff9e249c68123577958f1401a with delete_type TABLET_DATA_DELETED (replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:19.084517 23605 leader_election.cc:343] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.088402 23790 tablet_bootstrap.cc:492] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:19.088577 23790 ts_tablet_manager.cc:1403] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251023 20:15:19.089260 23790 raft_consensus.cc:359] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.089555 23790 raft_consensus.cc:385] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.089663 23790 raft_consensus.cc:740] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.090055 23783 ts_tablet_manager.cc:1929] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.090201 23783 log.cc:1199] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/487d9ee765d14aa2aa5668dc48fffe20
I20251023 20:15:19.090063 23790 consensus_queue.cc:260] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.090569 23783 ts_tablet_manager.cc:1950] T 487d9ee765d14aa2aa5668dc48fffe20 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.091548 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 487d9ee765d14aa2aa5668dc48fffe20 (table default.Table_561d6afe24b246c9909547a248a5fe7f [id=a0f9058297034e739a638f1747ddacfd]) successfully deleted
I20251023 20:15:19.092260 23787 tablet_bootstrap.cc:654] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.095381 23785 ts_tablet_manager.cc:1929] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.095538 23785 log.cc:1199] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/42ac1c6b30e048c58cfb7fc6ad0b5e69
I20251023 20:15:19.095920 23785 ts_tablet_manager.cc:1950] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:19.097013 23783 tablet_replica.cc:333] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.097204 23783 raft_consensus.cc:2243] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.097415 23783 raft_consensus.cc:2272] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.097612 23787 tablet_bootstrap.cc:492] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.097754 23787 ts_tablet_manager.cc:1403] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251023 20:15:19.098273 23783 ts_tablet_manager.cc:1916] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.098707 23787 raft_consensus.cc:359] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.098878 23787 raft_consensus.cc:385] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.098969 23787 raft_consensus.cc:740] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.099189 23787 consensus_queue.cc:260] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.101044 23787 ts_tablet_manager.cc:1434] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251023 20:15:19.101357 23787 tablet_bootstrap.cc:492] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.101554 23790 ts_tablet_manager.cc:1434] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251023 20:15:19.102406 23790 tablet_bootstrap.cc:492] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:19.103910 23787 tablet_bootstrap.cc:654] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.105958 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f2f55454e9b04d77860924e828522077" } new_table_name: "default.Table_29c36300df674fe88a11b50c46a26f27" modify_external_catalogs: true
I20251023 20:15:19.109293 24018 raft_consensus.cc:493] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.109458 24018 raft_consensus.cc:515] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.110076 24018 leader_election.cc:290] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.111436 23787 tablet_bootstrap.cc:492] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.111662 23787 ts_tablet_manager.cc:1403] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20251023 20:15:19.109109 23783 ts_tablet_manager.cc:1929] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.112524 23783 log.cc:1199] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/62acc395fb294965a1ab476d37f9fab1
I20251023 20:15:19.112993 23783 ts_tablet_manager.cc:1950] T 62acc395fb294965a1ab476d37f9fab1 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.114423 23787 raft_consensus.cc:359] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.114861 23787 raft_consensus.cc:385] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.114951 23787 raft_consensus.cc:740] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.114764 23783 tablet_replica.cc:333] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.115283 23783 raft_consensus.cc:2243] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.103941 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60348d0c00c14247860b9df14ddb82b9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.115648 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 62acc395fb294965a1ab476d37f9fab1 (table default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]) successfully deleted
I20251023 20:15:19.115715 23783 raft_consensus.cc:2272] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.115483 23787 consensus_queue.cc:260] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.115871 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 (table default.Table_3320634019ac4bf48649eff33160d808 [id=c06c5925ec4e418e91167cd090df7f81]) successfully deleted
I20251023 20:15:19.116431 23783 ts_tablet_manager.cc:1916] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.117345 23787 ts_tablet_manager.cc:1434] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:19.110955 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e61de7a93d450d9ed79eef82d7e5e3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.117990 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c95db7b5b9fd46acae8ffa7119915907" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:19.119441 23604 leader_election.cc:343] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.119589 23604 leader_election.cc:304] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:19.119825 23604 raft_consensus.cc:2686] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251023 20:15:19.120134 23604 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:19.120326 23604 leader_election.cc:343] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.120445 23604 leader_election.cc:304] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.121147 24018 raft_consensus.cc:2749] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:19.121850 23787 tablet_bootstrap.cc:492] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.122859 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e19f64a9ac44efba92928d0fa16819a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.125388 23787 tablet_bootstrap.cc:654] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.126148 23943 raft_consensus.cc:493] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.126309 23943 raft_consensus.cc:515] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.127014 23943 leader_election.cc:290] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:19.127216 23783 ts_tablet_manager.cc:1929] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.127365 23783 log.cc:1199] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/60348d0c00c14247860b9df14ddb82b9
I20251023 20:15:19.128108 23783 ts_tablet_manager.cc:1950] T 60348d0c00c14247860b9df14ddb82b9 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.128361 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.128800 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb6abe3ef848e2831839de6a28eb0f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:19.129370 23333 leader_election.cc:343] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.129977 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 60348d0c00c14247860b9df14ddb82b9 (table default.Table_baf231bb8eaa40eeb01169ddcf5161bd [id=94828b76eb8a4afdac4136fc626b6147]) successfully deleted
I20251023 20:15:19.128116 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb6abe3ef848e2831839de6a28eb0f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:19.130388 23670 raft_consensus.cc:2468] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:19.130824 23333 leader_election.cc:304] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.131016 23783 tablet_replica.cc:333] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.131213 23783 raft_consensus.cc:2243] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.131398 23783 raft_consensus.cc:2272] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.131883 23943 raft_consensus.cc:2804] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:19.131994 23943 raft_consensus.cc:493] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:19.132083 23943 raft_consensus.cc:3060] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.132865 23783 ts_tablet_manager.cc:1916] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.134443 23787 tablet_bootstrap.cc:492] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.134570 23787 ts_tablet_manager.cc:1403] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20251023 20:15:19.135071 23943 raft_consensus.cc:515] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.135310 23787 raft_consensus.cc:359] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.135483 23787 raft_consensus.cc:385] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.135562 23787 raft_consensus.cc:740] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.135767 23787 consensus_queue.cc:260] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.136507 23943 leader_election.cc:290] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:19.136924 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb6abe3ef848e2831839de6a28eb0f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:19.137805 23670 raft_consensus.cc:3060] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.138123 23787 ts_tablet_manager.cc:1434] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:19.138686 23787 tablet_bootstrap.cc:492] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.140554 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4bb6abe3ef848e2831839de6a28eb0f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:19.140974 23670 raft_consensus.cc:2468] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:19.141178 23333 leader_election.cc:343] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.141229 23787 tablet_bootstrap.cc:654] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.141443 23333 leader_election.cc:304] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.147845 23784 ts_tablet_manager.cc:1929] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.148010 23784 log.cc:1199] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/64f515399c494aadbed2bd0eac171a77
I20251023 20:15:19.148473 23784 ts_tablet_manager.cc:1950] T 64f515399c494aadbed2bd0eac171a77 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:19.148712 23783 ts_tablet_manager.cc:1929] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.149010 23783 log.cc:1199] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/42ac1c6b30e048c58cfb7fc6ad0b5e69
I20251023 20:15:19.149097 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 64f515399c494aadbed2bd0eac171a77 (table default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]) successfully deleted
I20251023 20:15:19.149515 23783 ts_tablet_manager.cc:1950] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:19.149832 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb915d114fc0421999d079c23cad8eb5 already in progress: creating tablet
I20251023 20:15:19.150281 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e61de7a93d450d9ed79eef82d7e5e3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.150822 23783 tablet_replica.cc:333] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.151204 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 (table default.Table_3320634019ac4bf48649eff33160d808 [id=c06c5925ec4e418e91167cd090df7f81]) successfully deleted
I20251023 20:15:19.151541 23943 raft_consensus.cc:2804] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
W20251023 20:15:19.151005 23605 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.151688 23787 tablet_bootstrap.cc:492] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.151923 23605 leader_election.cc:304] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.152001 23787 ts_tablet_manager.cc:1403] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20251023 20:15:19.152139 24018 raft_consensus.cc:2749] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:19.152454 23783 raft_consensus.cc:2243] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.152897 23783 raft_consensus.cc:2272] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.153129 23787 raft_consensus.cc:359] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.153374 23787 raft_consensus.cc:385] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.153491 23787 raft_consensus.cc:740] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.153787 23787 consensus_queue.cc:260] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.156004 23787 ts_tablet_manager.cc:1434] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251023 20:15:19.156818 23787 tablet_bootstrap.cc:492] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.159816 23787 tablet_bootstrap.cc:654] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:19.160476 24041 log.cc:927] Time spent T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20251023 20:15:19.161031 24037 tablet.cc:1722] T 34a738510f514b088469de3d8a39ca2c P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:19.153962 23783 ts_tablet_manager.cc:1916] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.162635 23943 raft_consensus.cc:697] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:19.169310 23143 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 7f766bfe7dd94352abcaf34a200d00bc (table default.Table_f2f55454e9b04d77860924e828522077 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 1 got 0
W20251023 20:15:19.170027 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f766bfe7dd94352abcaf34a200d00bc consensus state has no leader
I20251023 20:15:19.169960 23943 consensus_queue.cc:237] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.173563 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 33b5efb30b5c491c8eafe3ac25688843
I20251023 20:15:19.174314 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.175083 23783 ts_tablet_manager.cc:1929] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.175211 23783 log.cc:1199] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:19.175606 23783 ts_tablet_manager.cc:1950] T d0e61de7a93d450d9ed79eef82d7e5e3 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.175685 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.176246 23141 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 7f766bfe7dd94352abcaf34a200d00bc (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:19.176424 23141 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]). Expected version 2 got 0
I20251023 20:15:19.176568 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d0e61de7a93d450d9ed79eef82d7e5e3 (table default.Table_4d358925f1254129a8b42bfca11f8870 [id=a1701c81a217439197ef53ff39675ef9]) successfully deleted
I20251023 20:15:19.176604 23783 tablet_replica.cc:333] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.176793 23783 raft_consensus.cc:2243] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.176949 23783 raft_consensus.cc:2272] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:19.176992 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a636a4cdeb404e9f806d8e166ea57938 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a636a4cdeb404e9f806d8e166ea57938 already in progress: creating tablet
W20251023 20:15:19.177412 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 33b5efb30b5c491c8eafe3ac25688843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33b5efb30b5c491c8eafe3ac25688843 already in progress: creating tablet
I20251023 20:15:19.177726 23790 tablet_bootstrap.cc:654] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.179917 23783 ts_tablet_manager.cc:1916] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.183287 24020 mvcc.cc:204] Tried to move back new op lower bound from 7214082125703786496 to 7214082122030178304. Current Snapshot: MvccSnapshot[applied={T|T < 7214082125703786496}]
I20251023 20:15:19.195541 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 98a4553c1f014ef3ac975bf7a8076334 (DEFAULT_TABLE table=default.Table_ac4b704784f2423a82ad591bbed856ab [id=3cf9ba4080ce4e5ead110ed71e43ce1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.196012 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98a4553c1f014ef3ac975bf7a8076334. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.197726 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 83dae34129794f49b43038d3b1636ea0
I20251023 20:15:19.198843 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.202153 23637 tablet_service.cc:1505] Processing CreateTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 (DEFAULT_TABLE table=default.Table_ff42478856a6492aaade00ac66a02bb9 [id=04fe1d72c1d64f14a77cb4f66a6ad8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.202591 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb01b3f4ff7f41e98063ebf76abeace6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.204898 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 05933cb9433d48d285b2db82b289bdc8 (DEFAULT_TABLE table=default.Table_f5d04920d5d94f63a2f285d7d824d648 [id=9b498d31bbd945c391ffc62f4948ec85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.205299 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05933cb9433d48d285b2db82b289bdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.206993 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_49be7be29ce24a9ca4f5cc29644bc6c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.207428 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.208873 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.209071 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.209331 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_61af7ade632c46a3becb1573b6313f0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.209563 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:19.209869 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61af7ade632c46a3becb1573b6313f0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.210081 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f with delete_type TABLET_DATA_DELETED (replaced by tablet 8289ce37fa074891beb584bf2fff9747) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.210820 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 98a4553c1f014ef3ac975bf7a8076334 (DEFAULT_TABLE table=default.Table_ac4b704784f2423a82ad591bbed856ab [id=3cf9ba4080ce4e5ead110ed71e43ce1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.210995 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0d31043ac68d4a0394d7ba5b60d539c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.211215 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98a4553c1f014ef3ac975bf7a8076334. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:19.211402 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d31043ac68d4a0394d7ba5b60d539c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.212098 23783 ts_tablet_manager.cc:1929] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.212216 23783 log.cc:1199] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/caaccec5866a4e95afe9ccbf1f5bc494
I20251023 20:15:19.212441 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c04b1c255ffa45618b6c2c9a27eec857" } new_table_name: "default.Table_e168f64fab5849adb9cc390fffe62b57" modify_external_catalogs: true
I20251023 20:15:19.212564 23783 ts_tablet_manager.cc:1950] T caaccec5866a4e95afe9ccbf1f5bc494 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.213578 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_ff42478856a6492aaade00ac66a02bb9" } new_table_name: "default.Table_0843993aca2340378776a12d7b3a5d4c" modify_external_catalogs: true
I20251023 20:15:19.213745 23783 tablet_replica.cc:333] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.213980 23783 raft_consensus.cc:2243] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.214144 23783 raft_consensus.cc:2272] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.214644 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7a413e9be0f843bc97a50978fefdec72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.215072 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a413e9be0f843bc97a50978fefdec72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.216063 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8d8ada20a7bb43eb9a534aedca23cfd2" } modify_external_catalogs: true
I20251023 20:15:19.216221 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8d8ada20a7bb43eb9a534aedca23cfd2" } modify_external_catalogs: true
I20251023 20:15:19.219550 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 98a4553c1f014ef3ac975bf7a8076334 (DEFAULT_TABLE table=default.Table_ac4b704784f2423a82ad591bbed856ab [id=3cf9ba4080ce4e5ead110ed71e43ce1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.220002 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98a4553c1f014ef3ac975bf7a8076334. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.220805 23785 tablet_replica.cc:333] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:19.221007 23785 raft_consensus.cc:2243] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.221196 23785 raft_consensus.cc:2272] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.224404 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet caaccec5866a4e95afe9ccbf1f5bc494 (table default.Table_fb1a22feead44f33b0a8f5f5ec849493 [id=d05778ff4e54433d9d549b7d5356afe3]) successfully deleted
W20251023 20:15:19.224668 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a27f192792cf41288d69ca15ab538472 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a27f192792cf41288d69ca15ab538472 already in progress: creating tablet
W20251023 20:15:19.224902 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1ae910ac41fd4e349f159ab27e44375a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ae910ac41fd4e349f159ab27e44375a already in progress: creating tablet
I20251023 20:15:19.225140 23783 ts_tablet_manager.cc:1916] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.225768 23148 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:19.227409 23785 ts_tablet_manager.cc:1916] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.230846 23787 tablet_bootstrap.cc:492] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.231066 23787 ts_tablet_manager.cc:1403] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.075s user 0.034s sys 0.024s
I20251023 20:15:19.231904 23787 raft_consensus.cc:359] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.232151 23787 raft_consensus.cc:385] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.232295 23787 raft_consensus.cc:740] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.232622 23787 consensus_queue.cc:260] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.233851 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 05933cb9433d48d285b2db82b289bdc8 (DEFAULT_TABLE table=default.Table_f5d04920d5d94f63a2f285d7d824d648 [id=9b498d31bbd945c391ffc62f4948ec85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.234380 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05933cb9433d48d285b2db82b289bdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.234687 23783 ts_tablet_manager.cc:1929] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.233851 23368 tablet_service.cc:1505] Processing CreateTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 (DEFAULT_TABLE table=default.Table_ff42478856a6492aaade00ac66a02bb9 [id=04fe1d72c1d64f14a77cb4f66a6ad8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.239672 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet fe922c438a894a7aa04699b2d2897a28
I20251023 20:15:19.241367 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet cbacaf2fea24405caaf84d59b625fca3
I20251023 20:15:19.244323 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.244519 23783 log.cc:1199] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/2e338b982a564ea5b1b15c2146a96fa2
I20251023 20:15:19.244621 23943 raft_consensus.cc:493] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.244701 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb01b3f4ff7f41e98063ebf76abeace6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.245656 23783 ts_tablet_manager.cc:1950] T 2e338b982a564ea5b1b15c2146a96fa2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.242244 23787 ts_tablet_manager.cc:1434] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251023 20:15:19.244547 23941 raft_consensus.cc:493] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.245862 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.245921 23941 raft_consensus.cc:515] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.246165 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.246537 23637 tablet_service.cc:1505] Processing CreateTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c (DEFAULT_TABLE table=default.Table_a0edef92f99c4edebe22ae048c5dcdde [id=e0dec011bafc4166bdd335a6e94475d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.246556 23941 leader_election.cc:290] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.246883 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 19c58a406d564b048fb16a85aa0b754b (DEFAULT_TABLE table=default.Table_83bf80c4c34c4312be19e260564d4916 [id=83f2c96018ba47e79c3c7ad4df2852a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.246973 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bfbf0acdc84f2f8fe553694ee9897c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.246925 23646 tablet_service.cc:1505] Processing CreateTablet for tablet cbacaf2fea24405caaf84d59b625fca3 (DEFAULT_TABLE table=default.Table_8ff1c10b4f2f49dd950540c129481320 [id=4978d9ba6f66482e848efe9eb0c1c7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.247206 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c58a406d564b048fb16a85aa0b754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.247416 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbacaf2fea24405caaf84d59b625fca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.247459 23501 tablet_service.cc:1505] Processing CreateTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 (DEFAULT_TABLE table=default.Table_ff42478856a6492aaade00ac66a02bb9 [id=04fe1d72c1d64f14a77cb4f66a6ad8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.247895 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb01b3f4ff7f41e98063ebf76abeace6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.247989 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "040a44e272b7410c98c109fbceb19192" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:19.248481 23604 leader_election.cc:343] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:19.250164 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 124703edbf08408a9ca69732674c0ad3 consensus state has no leader
W20251023 20:15:19.250555 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb01b3f4ff7f41e98063ebf76abeace6 consensus state has no leader
I20251023 20:15:19.251107 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 5e73ae95c73945f081c298f49d47c904
I20251023 20:15:19.252979 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.238286 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.254235 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "040a44e272b7410c98c109fbceb19192" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.254946 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.255156 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 05933cb9433d48d285b2db82b289bdc8 (DEFAULT_TABLE table=default.Table_f5d04920d5d94f63a2f285d7d824d648 [id=9b498d31bbd945c391ffc62f4948ec85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.255527 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05933cb9433d48d285b2db82b289bdc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.256843 23943 raft_consensus.cc:515] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.257496 23399 raft_consensus.cc:2468] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:19.257688 23787 tablet_bootstrap.cc:492] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.258862 23605 leader_election.cc:304] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.259923 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:19.260247 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 33b5efb30b5c491c8eafe3ac25688843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33b5efb30b5c491c8eafe3ac25688843 already in progress: creating tablet
I20251023 20:15:19.260393 23369 tablet_service.cc:1505] Processing CreateTablet for tablet cbacaf2fea24405caaf84d59b625fca3 (DEFAULT_TABLE table=default.Table_8ff1c10b4f2f49dd950540c129481320 [id=4978d9ba6f66482e848efe9eb0c1c7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.260584 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 19c58a406d564b048fb16a85aa0b754b (DEFAULT_TABLE table=default.Table_83bf80c4c34c4312be19e260564d4916 [id=83f2c96018ba47e79c3c7ad4df2852a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.260818 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbacaf2fea24405caaf84d59b625fca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.261351 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c58a406d564b048fb16a85aa0b754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.263926 23377 tablet_service.cc:1505] Processing CreateTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c (DEFAULT_TABLE table=default.Table_a0edef92f99c4edebe22ae048c5dcdde [id=e0dec011bafc4166bdd335a6e94475d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.264325 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bfbf0acdc84f2f8fe553694ee9897c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.265885 23376 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.266456 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_3eb3339ecf9e490fa2c08cc6fe065029"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.266716 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a (DEFAULT_TABLE table=default.Table_42ca9b5fa333476f9d6a35fcb9d0249a [id=70a0aa0e964e425e8ee12c7205b494da]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:19.266983 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3eb3339ecf9e490fa2c08cc6fe065029 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.267100 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed0aa7e36e64843ad6b2ad4b415832a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.271234 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_578c34f1b0124b0296648c859e5904cb" } new_table_name: "default.Table_5d14af3d31254be5a7ff6cb5d1fea162" modify_external_catalogs: true
I20251023 20:15:19.272316 23363 tablet_service.cc:1505] Processing CreateTablet for tablet c05b9bcea6ff49188758271401270227 (DEFAULT_TABLE table=default.Table_f232c9e65a8248d2b818d81486618f5f [id=9834a6ff7acf4269aebcae4712fc7149]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.272680 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c05b9bcea6ff49188758271401270227. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.273386 23374 tablet_service.cc:1505] Processing CreateTablet for tablet eb5a6d9a4509431f97b69a3b138b5829 (DEFAULT_TABLE table=default.Table_96e1782778fe489e8919612f34dc710f [id=4b8e127e7d37433e8cfe495eda2bc815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.273732 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5a6d9a4509431f97b69a3b138b5829. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.274335 23943 leader_election.cc:290] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.276549 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7f5474847aa74113aca18fe96357be62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.277060 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f5474847aa74113aca18fe96357be62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.277745 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 124703edbf08408a9ca69732674c0ad3 (DEFAULT_TABLE table=default.Table_e168f64fab5849adb9cc390fffe62b57 [id=98abf33cd7324a51902de8bbf62de4b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.278115 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 124703edbf08408a9ca69732674c0ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.278259 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6179bdbf369e4b32be88ca151e71e50f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.278751 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6179bdbf369e4b32be88ca151e71e50f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.279254 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.279881 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.279829 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fa3c4b34d4744caf8f1fe02cf5b2326c" } new_table_name: "default.Table_16e3b6fb0a35485b81cea711419e6295" modify_external_catalogs: true
I20251023 20:15:19.281379 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_031b7f15cd9c4b17acd7c1c9c9102fbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.281839 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_031b7f15cd9c4b17acd7c1c9c9102fbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.283150 23786 tablet_bootstrap.cc:492] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:19.283360 23786 ts_tablet_manager.cc:1403] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.364s user 0.004s sys 0.012s
I20251023 20:15:19.283476 24037 tablet.cc:1722] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251023 20:15:19.284271 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c97e2e0f839a4ef389fe100b6a81301e consensus state has no leader
I20251023 20:15:19.286657 23941 raft_consensus.cc:2804] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:19.286851 23514 tablet_service.cc:1505] Processing CreateTablet for tablet cbacaf2fea24405caaf84d59b625fca3 (DEFAULT_TABLE table=default.Table_8ff1c10b4f2f49dd950540c129481320 [id=4978d9ba6f66482e848efe9eb0c1c7b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.289927 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 72408899f710470baaf6fdb268cb04c8 (DEFAULT_TABLE table=default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.290313 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72408899f710470baaf6fdb268cb04c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.295456 23361 tablet_service.cc:1505] Processing CreateTablet for tablet ac4d82d024fc491ab0dbe04147cb156c (DEFAULT_TABLE table=default.Table_baf578c1b5e748e09f244f10685992a8 [id=006294844eac4bcabdd6508738bdc4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.296291 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b98d2ec04db54e72a067731a38d21a56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.296847 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b98d2ec04db54e72a067731a38d21a56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.300113 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d82d024fc491ab0dbe04147cb156c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.300427 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b01089ea89ef4536a351098a4caa38f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.300984 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b01089ea89ef4536a351098a4caa38f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:19.303476 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0150ab3778c34c77a7e772dd6e82b32f consensus state has no leader
I20251023 20:15:19.307237 23941 raft_consensus.cc:493] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:19.307400 23941 raft_consensus.cc:3060] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.309643 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 124703edbf08408a9ca69732674c0ad3 (DEFAULT_TABLE table=default.Table_e168f64fab5849adb9cc390fffe62b57 [id=98abf33cd7324a51902de8bbf62de4b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.310070 23783 tablet_replica.cc:333] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.310098 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 124703edbf08408a9ca69732674c0ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.310002 23639 tablet_service.cc:1505] Processing CreateTablet for tablet eb5a6d9a4509431f97b69a3b138b5829 (DEFAULT_TABLE table=default.Table_96e1782778fe489e8919612f34dc710f [id=4b8e127e7d37433e8cfe495eda2bc815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.310155 23645 tablet_service.cc:1505] Processing CreateTablet for tablet ac4d82d024fc491ab0dbe04147cb156c (DEFAULT_TABLE table=default.Table_baf578c1b5e748e09f244f10685992a8 [id=006294844eac4bcabdd6508738bdc4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.310285 23783 raft_consensus.cc:2243] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.310463 23783 raft_consensus.cc:2272] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.310563 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5a6d9a4509431f97b69a3b138b5829. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.310778 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d82d024fc491ab0dbe04147cb156c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.311064 23783 ts_tablet_manager.cc:1916] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.313383 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 2e338b982a564ea5b1b15c2146a96fa2 (table default.Table_4f134fc02d1c4a53a726595803ef505c [id=ff3dd4d1748144c492b64ccde7297729]) successfully deleted
I20251023 20:15:19.316663 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a (DEFAULT_TABLE table=default.Table_42ca9b5fa333476f9d6a35fcb9d0249a [id=70a0aa0e964e425e8ee12c7205b494da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.317045 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed0aa7e36e64843ad6b2ad4b415832a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.317992 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbacaf2fea24405caaf84d59b625fca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.317154 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 72408899f710470baaf6fdb268cb04c8 (DEFAULT_TABLE table=default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.320699 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.325171 23515 tablet_service.cc:1505] Processing CreateTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c (DEFAULT_TABLE table=default.Table_a0edef92f99c4edebe22ae048c5dcdde [id=e0dec011bafc4166bdd335a6e94475d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.325575 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bfbf0acdc84f2f8fe553694ee9897c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.326424 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 19c58a406d564b048fb16a85aa0b754b (DEFAULT_TABLE table=default.Table_83bf80c4c34c4312be19e260564d4916 [id=83f2c96018ba47e79c3c7ad4df2852a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.326998 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c58a406d564b048fb16a85aa0b754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.327205 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a (DEFAULT_TABLE table=default.Table_42ca9b5fa333476f9d6a35fcb9d0249a [id=70a0aa0e964e425e8ee12c7205b494da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.327858 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "040a44e272b7410c98c109fbceb19192" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:19.328958 23644 tablet_service.cc:1505] Processing CreateTablet for tablet c05b9bcea6ff49188758271401270227 (DEFAULT_TABLE table=default.Table_f232c9e65a8248d2b818d81486618f5f [id=9834a6ff7acf4269aebcae4712fc7149]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.329463 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72408899f710470baaf6fdb268cb04c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.330880 23790 tablet_bootstrap.cc:492] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:19.331032 23790 ts_tablet_manager.cc:1403] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.229s user 0.005s sys 0.000s
I20251023 20:15:19.332945 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c05b9bcea6ff49188758271401270227. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.335065 23790 raft_consensus.cc:359] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.335251 23790 raft_consensus.cc:385] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.335328 23790 raft_consensus.cc:740] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.339221 23787 tablet_bootstrap.cc:654] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.349769 23783 ts_tablet_manager.cc:1929] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.349941 23783 log.cc:1199] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e4bb6abe3ef848e2831839de6a28eb0f
I20251023 20:15:19.350374 23783 ts_tablet_manager.cc:1950] T e4bb6abe3ef848e2831839de6a28eb0f P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:19.350940 23941 raft_consensus.cc:515] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.351286 23670 raft_consensus.cc:2393] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:19.352945 24080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.190:34331 every 8 connection(s)
I20251023 20:15:19.353749 23941 leader_election.cc:290] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.354981 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "040a44e272b7410c98c109fbceb19192" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:19.355254 23399 raft_consensus.cc:3060] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.355176 23790 consensus_queue.cc:260] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.356027 23790 ts_tablet_manager.cc:1434] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251023 20:15:19.356887 23790 tablet_bootstrap.cc:492] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:19.361119 23368 tablet_service.cc:1505] Processing CreateTablet for tablet d9f73477a06542c0b215ae86716cdc4e (DEFAULT_TABLE table=default.Table_79be15cb4350452b90277004ec803c8a [id=6604e04478d9438e990e19240dd10e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.361681 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f73477a06542c0b215ae86716cdc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.363325 23399 raft_consensus.cc:2468] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:19.364812 23642 tablet_service.cc:1505] Processing CreateTablet for tablet d9f73477a06542c0b215ae86716cdc4e (DEFAULT_TABLE table=default.Table_79be15cb4350452b90277004ec803c8a [id=6604e04478d9438e990e19240dd10e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.364874 23605 leader_election.cc:304] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:19.365335 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f73477a06542c0b215ae86716cdc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.368628 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.371266 23144 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]). Expected version 2 got 0
I20251023 20:15:19.374138 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet c95db7b5b9fd46acae8ffa7119915907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.375586 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 04f7dffd94f3442ea20045a512d7bd65 (DEFAULT_TABLE table=default.Table_3eb3339ecf9e490fa2c08cc6fe065029 [id=5bcb88a9719846c78bf5b7609e2bcb37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.376194 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f7dffd94f3442ea20045a512d7bd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.381003 23786 raft_consensus.cc:359] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.381302 23786 raft_consensus.cc:385] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.381433 23786 raft_consensus.cc:740] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.381779 23790 tablet_bootstrap.cc:654] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.381701 23786 consensus_queue.cc:260] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.382328 23786 ts_tablet_manager.cc:1434] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.099s user 0.002s sys 0.000s
I20251023 20:15:19.382673 23786 tablet_bootstrap.cc:492] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:19.382845 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed0aa7e36e64843ad6b2ad4b415832a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.383347 23827 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.190:34331
I20251023 20:15:19.390269 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 5407a850ec7045158f53cb8abf6fcb58 (DEFAULT_TABLE table=default.Table_61af7ade632c46a3becb1573b6313f0c [id=92ce6ecbc3fd4b48af6db9e587adcac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.390540 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 5407a850ec7045158f53cb8abf6fcb58 (DEFAULT_TABLE table=default.Table_61af7ade632c46a3becb1573b6313f0c [id=92ce6ecbc3fd4b48af6db9e587adcac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.390746 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5407a850ec7045158f53cb8abf6fcb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.390969 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5407a850ec7045158f53cb8abf6fcb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.390899 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "040a44e272b7410c98c109fbceb19192" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:19.391552 23604 leader_election.cc:343] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.393394 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 1edaed8c82da4e36a312477b15017460 (DEFAULT_TABLE table=default.Table_00e41d94627b471b9753905690b03a49 [id=f9b92376e02047ec9c965526e54739d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.393831 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edaed8c82da4e36a312477b15017460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.393991 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 1edaed8c82da4e36a312477b15017460 (DEFAULT_TABLE table=default.Table_00e41d94627b471b9753905690b03a49 [id=f9b92376e02047ec9c965526e54739d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.394356 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edaed8c82da4e36a312477b15017460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.396144 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 514cb3b1ed024041ac160a7d9acff618 (DEFAULT_TABLE table=default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.396567 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514cb3b1ed024041ac160a7d9acff618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.397536 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 385900c60d584c15b702f186ecfbc036 (DEFAULT_TABLE table=default.Table_49f2383c82e44457827030102d18d5bb [id=f49d21d543f5498580620fda6cfc97e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.397805 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 385900c60d584c15b702f186ecfbc036 (DEFAULT_TABLE table=default.Table_49f2383c82e44457827030102d18d5bb [id=f49d21d543f5498580620fda6cfc97e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.398154 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385900c60d584c15b702f186ecfbc036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.398264 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385900c60d584c15b702f186ecfbc036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.403825 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 6e4189623c7a4ca1bdf73f71663e2b77 (DEFAULT_TABLE table=default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 [id=d236ba1e610441a8886230b0365ee300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.404084 23647 tablet_service.cc:1505] Processing CreateTablet for tablet a570f339890946b7ab20c15de68462d5 (DEFAULT_TABLE table=default.Table_a87b53598d1a4c0b9324ab792449a67d [id=ff726ba9ad7c4b6491970a8c0cd2e42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.403888 23643 tablet_service.cc:1505] Processing CreateTablet for tablet bb475e303d07491bbc19dc3f628bceb7 (DEFAULT_TABLE table=default.Table_0d31043ac68d4a0394d7ba5b60d539c8 [id=0a9d3a3303d945f199d265637a859772]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.404340 23640 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1bf4f01fbc4645bcf8392f56f853b2 (DEFAULT_TABLE table=default.Table_6de6ff0130d74ad894cfa487113e1b19 [id=7a4098cc9d4e49a39ef792883f85d564]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.404654 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4189623c7a4ca1bdf73f71663e2b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.404918 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb475e303d07491bbc19dc3f628bceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.406002 23503 tablet_service.cc:1505] Processing CreateTablet for tablet 72408899f710470baaf6fdb268cb04c8 (DEFAULT_TABLE table=default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.408439 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1bf4f01fbc4645bcf8392f56f853b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.404915 23501 tablet_service.cc:1505] Processing CreateTablet for tablet c05b9bcea6ff49188758271401270227 (DEFAULT_TABLE table=default.Table_f232c9e65a8248d2b818d81486618f5f [id=9834a6ff7acf4269aebcae4712fc7149]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.411209 23508 tablet_service.cc:1505] Processing CreateTablet for tablet ac4d82d024fc491ab0dbe04147cb156c (DEFAULT_TABLE table=default.Table_baf578c1b5e748e09f244f10685992a8 [id=006294844eac4bcabdd6508738bdc4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.411723 23514 tablet_service.cc:1505] Processing CreateTablet for tablet eb5a6d9a4509431f97b69a3b138b5829 (DEFAULT_TABLE table=default.Table_96e1782778fe489e8919612f34dc710f [id=4b8e127e7d37433e8cfe495eda2bc815]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.412196 23512 tablet_service.cc:1505] Processing CreateTablet for tablet 6e4189623c7a4ca1bdf73f71663e2b77 (DEFAULT_TABLE table=default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 [id=d236ba1e610441a8886230b0365ee300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.412235 23516 tablet_service.cc:1505] Processing CreateTablet for tablet d9f73477a06542c0b215ae86716cdc4e (DEFAULT_TABLE table=default.Table_79be15cb4350452b90277004ec803c8a [id=6604e04478d9438e990e19240dd10e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.412344 23941 raft_consensus.cc:2804] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:19.412736 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 1edaed8c82da4e36a312477b15017460 (DEFAULT_TABLE table=default.Table_00e41d94627b471b9753905690b03a49 [id=f9b92376e02047ec9c965526e54739d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.412674 23941 raft_consensus.cc:697] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:19.413425 23502 tablet_service.cc:1505] Processing CreateTablet for tablet 04f7dffd94f3442ea20045a512d7bd65 (DEFAULT_TABLE table=default.Table_3eb3339ecf9e490fa2c08cc6fe065029 [id=5bcb88a9719846c78bf5b7609e2bcb37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.413604 23941 consensus_queue.cc:237] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.415565 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a570f339890946b7ab20c15de68462d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.415577 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 8a630b83f614447ebf2a08851d6b827d (DEFAULT_TABLE table=default.Table_d39a64892c5b49d99dccfbe4163d6f53 [id=568404f36000475ab792fa4321f99249]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.417675 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet e4bb6abe3ef848e2831839de6a28eb0f (table default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]) successfully deleted
I20251023 20:15:19.418454 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 124703edbf08408a9ca69732674c0ad3 (DEFAULT_TABLE table=default.Table_c04b1c255ffa45618b6c2c9a27eec857 [id=98abf33cd7324a51902de8bbf62de4b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.421661 23509 tablet_service.cc:1505] Processing CreateTablet for tablet 514cb3b1ed024041ac160a7d9acff618 (DEFAULT_TABLE table=default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.421845 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72408899f710470baaf6fdb268cb04c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.422072 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f73477a06542c0b215ae86716cdc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.422289 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 8a630b83f614447ebf2a08851d6b827d (DEFAULT_TABLE table=default.Table_d39a64892c5b49d99dccfbe4163d6f53 [id=568404f36000475ab792fa4321f99249]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.422722 23513 tablet_service.cc:1505] Processing CreateTablet for tablet bb475e303d07491bbc19dc3f628bceb7 (DEFAULT_TABLE table=default.Table_0d31043ac68d4a0394d7ba5b60d539c8 [id=0a9d3a3303d945f199d265637a859772]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.355001 23785 ts_tablet_manager.cc:1929] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.428184 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a630b83f614447ebf2a08851d6b827d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.428407 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 04f7dffd94f3442ea20045a512d7bd65 (DEFAULT_TABLE table=default.Table_3eb3339ecf9e490fa2c08cc6fe065029 [id=5bcb88a9719846c78bf5b7609e2bcb37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.429741 23500 tablet_service.cc:1505] Processing CreateTablet for tablet 385900c60d584c15b702f186ecfbc036 (DEFAULT_TABLE table=default.Table_49f2383c82e44457827030102d18d5bb [id=f49d21d543f5498580620fda6cfc97e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.431133 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c05b9bcea6ff49188758271401270227. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.432958 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1edaed8c82da4e36a312477b15017460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.433204 23506 tablet_service.cc:1505] Processing CreateTablet for tablet 5407a850ec7045158f53cb8abf6fcb58 (DEFAULT_TABLE table=default.Table_61af7ade632c46a3becb1573b6313f0c [id=92ce6ecbc3fd4b48af6db9e587adcac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.432825 23783 tablet_replica.cc:333] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.433673 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3eb3339ecf9e490fa2c08cc6fe065029" } new_table_name: "default.Table_19b247b5c09d421dbb76b9cbbd739309" modify_external_catalogs: true
I20251023 20:15:19.434094 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 6e4189623c7a4ca1bdf73f71663e2b77 (DEFAULT_TABLE table=default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 [id=d236ba1e610441a8886230b0365ee300]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.434528 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4189623c7a4ca1bdf73f71663e2b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.436760 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f7dffd94f3442ea20045a512d7bd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.437218 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0" } modify_external_catalogs: true
I20251023 20:15:19.437438 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0" } modify_external_catalogs: true
I20251023 20:15:19.439582 23827 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-0/data/info.pb
I20251023 20:15:19.439879 23369 tablet_service.cc:1505] Processing CreateTablet for tablet bb475e303d07491bbc19dc3f628bceb7 (DEFAULT_TABLE table=default.Table_0d31043ac68d4a0394d7ba5b60d539c8 [id=0a9d3a3303d945f199d265637a859772]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.440357 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb475e303d07491bbc19dc3f628bceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.441994 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 514cb3b1ed024041ac160a7d9acff618 (DEFAULT_TABLE table=default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.442396 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514cb3b1ed024041ac160a7d9acff618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.444662 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 23827
I20251023 20:15:19.448719 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 8a630b83f614447ebf2a08851d6b827d (DEFAULT_TABLE table=default.Table_d39a64892c5b49d99dccfbe4163d6f53 [id=568404f36000475ab792fa4321f99249]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.449154 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a630b83f614447ebf2a08851d6b827d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.450379 23377 tablet_service.cc:1505] Processing CreateTablet for tablet a570f339890946b7ab20c15de68462d5 (DEFAULT_TABLE table=default.Table_a87b53598d1a4c0b9324ab792449a67d [id=ff726ba9ad7c4b6491970a8c0cd2e42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.450765 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a570f339890946b7ab20c15de68462d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.454300 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fc18b4fbf71a45f6b3cfee31effd8eef" } modify_external_catalogs: true
I20251023 20:15:19.454522 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fc18b4fbf71a45f6b3cfee31effd8eef" } modify_external_catalogs: true
I20251023 20:15:19.456053 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_39094702f1ba4174a5eb05a40213e745" } modify_external_catalogs: true
I20251023 20:15:19.456231 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_39094702f1ba4174a5eb05a40213e745" } modify_external_catalogs: true
W20251023 20:15:19.459023 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04f7dffd94f3442ea20045a512d7bd65 consensus state has no leader
I20251023 20:15:19.460621 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 1a64a7c2e1f242e1b54243489622bae5
I20251023 20:15:19.464502 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a21ec49194e24723b31fd9c86f1f85e4
I20251023 20:15:19.467185 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 7301d6121c9d4c8dbc642e895aa4e9f6
I20251023 20:15:19.470063 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7908a191ee99400093d9f2deb05e52d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.470326 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e926815d67914fed915b184838f05b72" } modify_external_catalogs: true
I20251023 20:15:19.470515 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e926815d67914fed915b184838f05b72" } modify_external_catalogs: true
W20251023 20:15:19.470772 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7908a191ee99400093d9f2deb05e52d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.472013 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7c7ed77bec2746e0ba85dd8598b4c364" } new_table_name: "default.Table_80e9f701dbaf4b009a1232cce62e2741" modify_external_catalogs: true
I20251023 20:15:19.472222 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7d29445c967d4d7fb9d42c9d178c677b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.472711 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d29445c967d4d7fb9d42c9d178c677b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.474927 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b93a09969d41431aa77623ab62d6f6e7" } new_table_name: "default.Table_bd447f978d1d4b8bad839b915907db66" modify_external_catalogs: true
I20251023 20:15:19.481019 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_87caa0c69120413a92c6b65337baad14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.481702 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87caa0c69120413a92c6b65337baad14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:19.483721 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5368813164949ff80e27bf59dcc931a consensus state has no leader
I20251023 20:15:19.442672 23498 tablet_service.cc:1505] Processing CreateTablet for tablet a570f339890946b7ab20c15de68462d5 (DEFAULT_TABLE table=default.Table_a87b53598d1a4c0b9324ab792449a67d [id=ff726ba9ad7c4b6491970a8c0cd2e42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.485738 23145 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fab9daaa8e644d2a9681e2229696c2bb" } modify_external_catalogs: true
I20251023 20:15:19.485991 23145 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fab9daaa8e644d2a9681e2229696c2bb" } modify_external_catalogs: true
I20251023 20:15:19.488022 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d82d024fc491ab0dbe04147cb156c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.490268 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6714c0ab9081464a91ac8bdb8eab9107"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.490831 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6714c0ab9081464a91ac8bdb8eab9107 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.491732 23512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4189623c7a4ca1bdf73f71663e2b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.494779 23786 tablet_bootstrap.cc:654] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.498097 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_86168bc39ef14e358f5356df1fe2e346"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.498945 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86168bc39ef14e358f5356df1fe2e346 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.508867 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_240b16ca6e1543f2aab607c1d6a447fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.509516 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_240b16ca6e1543f2aab607c1d6a447fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.499022 23513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb475e303d07491bbc19dc3f628bceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.500582 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "040a44e272b7410c98c109fbceb19192" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.510860 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a6f1c53dfe964d538a77e7b6570d9a9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.511371 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6f1c53dfe964d538a77e7b6570d9a9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.511649 23498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a570f339890946b7ab20c15de68462d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.501375 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1bf4f01fbc4645bcf8392f56f853b2 (DEFAULT_TABLE table=default.Table_6de6ff0130d74ad894cfa487113e1b19 [id=7a4098cc9d4e49a39ef792883f85d564]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.501212 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (DEFAULT_TABLE table=default.Table_7c5125a919774476a6ec9f806c2d329e [id=98e1db67295f441c9904194cba2da98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.514055 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514cb3b1ed024041ac160a7d9acff618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.515511 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5407a850ec7045158f53cb8abf6fcb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.518879 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f7dffd94f3442ea20045a512d7bd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.520524 23500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385900c60d584c15b702f186ecfbc036. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.521826 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a630b83f614447ebf2a08851d6b827d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.522647 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 975fc4848c8f4db4bfd2dad9ac620d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.527060 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 124703edbf08408a9ca69732674c0ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.530867 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1bf4f01fbc4645bcf8392f56f853b2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:19.530411 23134 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_01e28a3f1691446384d30911e4894273 [id=b0c6ab2358b84be2862d31f863e3c32d], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9e76f2e26dcd4cdfbf72de899f5a1458 consensus state has no leader
I20251023 20:15:19.539646 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5a6d9a4509431f97b69a3b138b5829. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:19.541484 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 4b6c3e7169154e3497dd3570dca0d745 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b6c3e7169154e3497dd3570dca0d745 already in progress: creating tablet
I20251023 20:15:19.498068 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e168f64fab5849adb9cc390fffe62b57" } new_table_name: "default.Table_c07286b67c934b439529460275aa2631" modify_external_catalogs: true
I20251023 20:15:19.545639 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 2a1bf4f01fbc4645bcf8392f56f853b2 (DEFAULT_TABLE table=default.Table_6de6ff0130d74ad894cfa487113e1b19 [id=7a4098cc9d4e49a39ef792883f85d564]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.549702 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d698ffe7d154ef8b1994ffdd4b0c648 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.550021 23784 tablet_replica.cc:333] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:19.550233 23784 raft_consensus.cc:2243] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.550442 23784 raft_consensus.cc:2272] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.551105 23784 ts_tablet_manager.cc:1916] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.551911 24018 raft_consensus.cc:493] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.553153 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.553444 23942 raft_consensus.cc:493] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.552112 24018 raft_consensus.cc:515] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.554206 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.554769 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1bf4f01fbc4645bcf8392f56f853b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.554803 23790 tablet_bootstrap.cc:492] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:19.555315 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.556618 23790 ts_tablet_manager.cc:1403] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.198s user 0.005s sys 0.000s
W20251023 20:15:19.557336 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 83dae34129794f49b43038d3b1636ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83dae34129794f49b43038d3b1636ea0 already in progress: creating tablet
I20251023 20:15:19.557555 23790 raft_consensus.cc:359] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:19.557688 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fe922c438a894a7aa04699b2d2897a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe922c438a894a7aa04699b2d2897a28 already in progress: creating tablet
I20251023 20:15:19.557786 23790 raft_consensus.cc:385] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.557854 23790 raft_consensus.cc:740] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.558112 23790 consensus_queue.cc:260] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.558583 23785 log.cc:1199] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/caaccec5866a4e95afe9ccbf1f5bc494
I20251023 20:15:19.559069 23785 ts_tablet_manager.cc:1950] T caaccec5866a4e95afe9ccbf1f5bc494 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:19.559307 23333 leader_election.cc:343] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.554855 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.560993 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.561841 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e31aebfc87734ffeb2c0b25c07cb3f70" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:19.557961 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cbacaf2fea24405caaf84d59b625fca3 because the tablet was not found. No further retry: Not found: Tablet not found: cbacaf2fea24405caaf84d59b625fca3
W20251023 20:15:19.562440 23604 leader_election.cc:343] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:19.563071 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5e73ae95c73945f081c298f49d47c904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e73ae95c73945f081c298f49d47c904 already in progress: creating tablet
I20251023 20:15:19.573563 23373 tablet_service.cc:1505] Processing CreateTablet for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (DEFAULT_TABLE table=default.Table_7c5125a919774476a6ec9f806c2d329e [id=98e1db67295f441c9904194cba2da98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.574393 24018 leader_election.cc:290] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.574769 23942 raft_consensus.cc:515] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.575157 23649 tablet_service.cc:1505] Processing CreateTablet for tablet b90810781f724c78935a33d868ffa27d (DEFAULT_TABLE table=default.Table_fab9daaa8e644d2a9681e2229696c2bb [id=2df3ca33fcf54a978b31fc24b9d21ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.575443 23942 leader_election.cc:290] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.575573 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90810781f724c78935a33d868ffa27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.575596 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.575783 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.576620 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585a9af140d9479fb83d13e68a62300c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.578215 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_365837ad05624d75855461852362db1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.578712 23604 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:19.578790 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_365837ad05624d75855461852362db1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.587059 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (DEFAULT_TABLE table=default.Table_7c5125a919774476a6ec9f806c2d329e [id=98e1db67295f441c9904194cba2da98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.587563 23783 raft_consensus.cc:2243] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.587852 23783 raft_consensus.cc:2272] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.587935 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 975fc4848c8f4db4bfd2dad9ac620d83. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:19.593447 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a54470d088748b0be22ab9db5a25f87 consensus state has no leader
I20251023 20:15:19.594512 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0
I20251023 20:15:19.595286 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet 83da09ee0d10435ba89a19dc378d34eb from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:19.607612 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d491cd3951344335be8170dcdcb94c4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.604532 23516 tablet_service.cc:1505] Processing CreateTablet for tablet b90810781f724c78935a33d868ffa27d (DEFAULT_TABLE table=default.Table_fab9daaa8e644d2a9681e2229696c2bb [id=2df3ca33fcf54a978b31fc24b9d21ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.608162 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_914e9b5e4a7443c5baf677b4b6857359" } new_table_name: "default.Table_54bd013a97df4bb08fdab2eeea9f5d42" modify_external_catalogs: true
W20251023 20:15:19.608304 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d491cd3951344335be8170dcdcb94c4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.609930 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_efda7d26e2124ec4b1eeca32fe1b545e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.610690 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 975fc4848c8f4db4bfd2dad9ac620d83. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:19.610689 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efda7d26e2124ec4b1eeca32fe1b545e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.611241 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet e0f8d8184a68453abd2792ccc7740aa1 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:19.612007 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.614605 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90810781f724c78935a33d868ffa27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.621245 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585a9af140d9479fb83d13e68a62300c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.622401 23147 master_service.cc:954] ReplaceTablet: received request to replace tablet 25079532547d4175802b84877790662d from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:19.621614 23333 leader_election.cc:304] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.625733 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e31aebfc87734ffeb2c0b25c07cb3f70" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.628098 23148 master_service.cc:954] ReplaceTablet: received request to replace tablet 6a54470d088748b0be22ab9db5a25f87 from {username='slave'} at 127.0.0.1:40384
W20251023 20:15:19.639472 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 124703edbf08408a9ca69732674c0ad3 consensus state has no leader
I20251023 20:15:19.649034 23445 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
W20251023 20:15:19.662571 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5346a221c78342359eb6e418bd12cd24 consensus state has no leader
I20251023 20:15:19.663573 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a216eab77a9f465982dad2f085170ddf" } new_table_name: "default.Table_3f6d1412904e4106b793f17c29014aec" modify_external_catalogs: true
I20251023 20:15:19.665742 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.666066 24018 raft_consensus.cc:493] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.669281 23942 raft_consensus.cc:493] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.669421 23942 raft_consensus.cc:515] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.670220 23942 leader_election.cc:290] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.670499 24018 raft_consensus.cc:515] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.671027 24018 leader_election.cc:290] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:19.671231 23605 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.671345 23605 leader_election.cc:304] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:19.671659 23605 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 49148.6us
W20251023 20:15:19.672309 23605 leader_election.cc:343] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.672438 23605 leader_election.cc:304] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.673185 24018 raft_consensus.cc:2749] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:19.673357 24018 raft_consensus.cc:2749] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:19.673828 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6714c0ab9081464a91ac8bdb8eab9107" } modify_external_catalogs: true
I20251023 20:15:19.674036 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6714c0ab9081464a91ac8bdb8eab9107" } modify_external_catalogs: true
I20251023 20:15:19.674358 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bcdec7c16c943079e20bb486aecb431" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.674582 23398 raft_consensus.cc:2468] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:19.674964 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.675144 23399 raft_consensus.cc:2468] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:19.676685 23605 leader_election.cc:304] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:19.677117 23605 leader_election.cc:304] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:19.677522 24018 raft_consensus.cc:2804] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:19.677665 24018 raft_consensus.cc:493] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:19.677662 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet b90810781f724c78935a33d868ffa27d
I20251023 20:15:19.677853 24018 raft_consensus.cc:3060] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.680648 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.681280 23942 raft_consensus.cc:2804] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:19.681449 23942 raft_consensus.cc:493] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:19.681592 23942 raft_consensus.cc:3060] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.685774 23783 ts_tablet_manager.cc:1916] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.689487 23942 raft_consensus.cc:515] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.692544 23145 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_da2bae6bea6047318cd9fb653b35475c" } modify_external_catalogs: true
I20251023 20:15:19.692796 23145 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_da2bae6bea6047318cd9fb653b35475c" } modify_external_catalogs: true
I20251023 20:15:19.693678 23942 leader_election.cc:290] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.693711 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 25079532547d4175802b84877790662d
I20251023 20:15:19.694020 24018 raft_consensus.cc:515] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.694208 23147 catalog_manager.cc:6719] ReplaceTablet: tablet 25079532547d4175802b84877790662d deleted and replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf
I20251023 20:15:19.694530 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:19.694911 23148 catalog_manager.cc:6719] ReplaceTablet: tablet 6a54470d088748b0be22ab9db5a25f87 deleted and replaced by tablet 3650822dd30c4c8e97982636ac692eae
I20251023 20:15:19.695139 24018 leader_election.cc:290] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.695218 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_54bd013a97df4bb08fdab2eeea9f5d42" } modify_external_catalogs: true
I20251023 20:15:19.695367 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_54bd013a97df4bb08fdab2eeea9f5d42" } modify_external_catalogs: true
I20251023 20:15:19.698920 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_09442523fb5d4e9d856cdcfba51fb911" } new_table_name: "default.Table_97743c4429454792b72a02f5fa2c3123" modify_external_catalogs: true
I20251023 20:15:19.700194 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_00f072ef064344c48ccd7c0982707e50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.700738 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00f072ef064344c48ccd7c0982707e50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.700883 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.702934 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bcdec7c16c943079e20bb486aecb431" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:19.703123 23398 raft_consensus.cc:3060] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.705667 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:19.705857 23399 raft_consensus.cc:3060] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:19.705886 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:19.706561 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:19.707890 23604 leader_election.cc:343] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.708213 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bcdec7c16c943079e20bb486aecb431" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.708472 23399 raft_consensus.cc:2468] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
W20251023 20:15:19.708756 23604 leader_election.cc:343] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.709156 23605 leader_election.cc:304] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.709604 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.710096 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.711410 23398 raft_consensus.cc:2468] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:19.711477 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.711766 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bcdec7c16c943079e20bb486aecb431" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:19.712003 23604 leader_election.cc:343] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:19.712543 23604 leader_election.cc:343] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.712839 23605 leader_election.cc:304] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:19.714272 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2a27706ecdb4ab78bb96c10a9f6757f consensus state has no leader
I20251023 20:15:19.714963 24018 raft_consensus.cc:2804] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:19.716207 24018 raft_consensus.cc:697] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:19.716293 23786 tablet_bootstrap.cc:492] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:19.717193 23942 raft_consensus.cc:2804] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:19.716822 24018 consensus_queue.cc:237] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.717936 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.721472 24087 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:15:19.723778 23716 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
I20251023 20:15:19.726941 24087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Bootstrap starting.
I20251023 20:15:19.727988 23140 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 0c577bc5b1dc40c395d45878587b6dc0
I20251023 20:15:19.730736 23786 ts_tablet_manager.cc:1403] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.334s user 0.006s sys 0.000s
I20251023 20:15:19.732739 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 925e53d9ea6f4c7baa38551eb917828d
I20251023 20:15:19.734045 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.740837 23145 master_service.cc:954] ReplaceTablet: received request to replace tablet 7540354f8577482785e6341fa9022315 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:19.741281 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 5346a221c78342359eb6e418bd12cd24
I20251023 20:15:19.743494 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:19.744258 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9c5b69df4c04365b1ac1457fe8300cc consensus state has no leader
I20251023 20:15:19.746217 24087 log.cc:826] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:19.749542 23790 ts_tablet_manager.cc:1434] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.193s user 0.002s sys 0.000s
I20251023 20:15:19.749944 23790 tablet_bootstrap.cc:492] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:19.751856 23790 tablet_bootstrap.cc:654] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:19.751883 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 201028us
I20251023 20:15:19.752983 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.753604 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.753729 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.753849 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.753979 23943 raft_consensus.cc:2749] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:19.753737 23361 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.754134 23943 raft_consensus.cc:493] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.754258 23943 raft_consensus.cc:515] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.754937 23943 leader_election.cc:290] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.756937 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.761268 23363 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.761796 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.761992 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:19.762327 23377 tablet_service.cc:1505] Processing CreateTablet for tablet b90810781f724c78935a33d868ffa27d (DEFAULT_TABLE table=default.Table_fab9daaa8e644d2a9681e2229696c2bb [id=2df3ca33fcf54a978b31fc24b9d21ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.762981 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90810781f724c78935a33d868ffa27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.764722 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.765882 23445 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
I20251023 20:15:19.767668 23582 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
W20251023 20:15:19.768870 23333 leader_election.cc:343] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.772778 23787 tablet_bootstrap.cc:492] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.772964 23787 ts_tablet_manager.cc:1403] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.519s user 0.014s sys 0.061s
I20251023 20:15:19.774042 23787 raft_consensus.cc:359] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.774263 23787 raft_consensus.cc:385] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.774338 23787 raft_consensus.cc:740] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.778378 23786 raft_consensus.cc:359] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.779022 23786 raft_consensus.cc:385] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.779160 23786 raft_consensus.cc:740] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.779487 23786 consensus_queue.cc:260] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.780237 23786 ts_tablet_manager.cc:1434] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20251023 20:15:19.785122 23582 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
I20251023 20:15:19.781999 23786 tablet_bootstrap.cc:492] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:19.787358 23787 consensus_queue.cc:260] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.791350 23784 ts_tablet_manager.cc:1929] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.792964 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_25427c3d36fb4ba4a01ac362787c81b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.793218 23784 log.cc:1199] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/2d698ffe7d154ef8b1994ffdd4b0c648
W20251023 20:15:19.793619 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25427c3d36fb4ba4a01ac362787c81b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.795488 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 487d9ee765d14aa2aa5668dc48fffe20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.796312 23787 ts_tablet_manager.cc:1434] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20251023 20:15:19.796444 23333 leader_election.cc:304] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.797923 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_39fdd0d19cae4e928fdee07e74900b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.798285 23784 ts_tablet_manager.cc:1950] T 2d698ffe7d154ef8b1994ffdd4b0c648 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:19.798482 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39fdd0d19cae4e928fdee07e74900b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.799224 23787 tablet_bootstrap.cc:492] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.799810 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_43dc7ae468fd42cb867849c762736c66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.800089 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 2d698ffe7d154ef8b1994ffdd4b0c648 (table default.Table_ea61082402e14334acc5162a368129e3 [id=1a48facca0024a5eaa670799650b9d6d]) successfully deleted
W20251023 20:15:19.800307 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43dc7ae468fd42cb867849c762736c66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.800841 23787 tablet_bootstrap.cc:654] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.801316 23786 tablet_bootstrap.cc:654] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.802119 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_83bf80c4c34c4312be19e260564d4916" } new_table_name: "default.Table_ad94afb284cc49b8ab09122184009906" modify_external_catalogs: true
I20251023 20:15:19.803448 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a0edef92f99c4edebe22ae048c5dcdde" } modify_external_catalogs: true
I20251023 20:15:19.803642 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a0edef92f99c4edebe22ae048c5dcdde" } modify_external_catalogs: true
W20251023 20:15:19.805157 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1a64a7c2e1f242e1b54243489622bae5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a64a7c2e1f242e1b54243489622bae5 already in progress: creating tablet
W20251023 20:15:19.806584 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a21ec49194e24723b31fd9c86f1f85e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a21ec49194e24723b31fd9c86f1f85e4 already in progress: creating tablet
W20251023 20:15:19.807530 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7301d6121c9d4c8dbc642e895aa4e9f6 already in progress: creating tablet
W20251023 20:15:19.808337 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 already in progress: creating tablet
W20251023 20:15:19.809159 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6a54470d088748b0be22ab9db5a25f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a54470d088748b0be22ab9db5a25f87 already in progress: creating tablet
W20251023 20:15:19.809903 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
I20251023 20:15:19.811100 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 7540354f8577482785e6341fa9022315
W20251023 20:15:19.811431 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 25079532547d4175802b84877790662d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25079532547d4175802b84877790662d already in progress: creating tablet
I20251023 20:15:19.811650 23145 catalog_manager.cc:6719] ReplaceTablet: tablet 7540354f8577482785e6341fa9022315 deleted and replaced by tablet e194d03a6aa946fbb2c79d91b196440b
I20251023 20:15:19.812340 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.812798 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.813154 23783 ts_tablet_manager.cc:1929] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:19.813225 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 925e53d9ea6f4c7baa38551eb917828d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 925e53d9ea6f4c7baa38551eb917828d already in progress: creating tablet
I20251023 20:15:19.813402 23783 log.cc:1199] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/c95db7b5b9fd46acae8ffa7119915907
I20251023 20:15:19.813834 23783 ts_tablet_manager.cc:1950] T c95db7b5b9fd46acae8ffa7119915907 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:19.813977 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5346a221c78342359eb6e418bd12cd24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5346a221c78342359eb6e418bd12cd24 already in progress: creating tablet
I20251023 20:15:19.814201 23784 tablet_replica.cc:333] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:19.815485 23784 raft_consensus.cc:2243] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.815908 23784 raft_consensus.cc:2272] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.816267 24087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=23 ignored=0} mutations{seen=25 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:19.817042 24087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: Bootstrap complete.
I20251023 20:15:19.817250 23784 ts_tablet_manager.cc:1916] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.817967 23943 raft_consensus.cc:2749] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:19.819347 23373 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.819615 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.819924 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.820839 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c37d72052930485b819533573f3710ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.821456 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c37d72052930485b819533573f3710ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.825927 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet caaccec5866a4e95afe9ccbf1f5bc494 (table default.Table_fb1a22feead44f33b0a8f5f5ec849493 [id=d05778ff4e54433d9d549b7d5356afe3]) successfully deleted
I20251023 20:15:19.826045 23785 tablet_replica.cc:333] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:19.825167 24087 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:19.826229 24087 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6545140d94b7485aa81c51ad233355cd, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.827497 24087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:19.828473 23786 tablet_bootstrap.cc:492] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:19.828512 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c760c475c8554c89aeb6fd77fd422797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.828804 23786 ts_tablet_manager.cc:1403] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
W20251023 20:15:19.829181 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c760c475c8554c89aeb6fd77fd422797 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.831063 24087 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:19.830900 23786 raft_consensus.cc:359] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.832623 23786 raft_consensus.cc:385] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.832722 23786 raft_consensus.cc:740] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.833204 23786 consensus_queue.cc:260] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.837474 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_baf8abd9a132412a8b53ac8d9e044dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.838121 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baf8abd9a132412a8b53ac8d9e044dcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.838876 23786 ts_tablet_manager.cc:1434] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251023 20:15:19.841388 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_19b247b5c09d421dbb76b9cbbd739309" } new_table_name: "default.Table_b827c36444e34ec6924cfb0ac859b3e1" modify_external_catalogs: true
I20251023 20:15:19.841547 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet c95db7b5b9fd46acae8ffa7119915907 (table default.Table_91596859556249fbb6e82b7dc5756a92 [id=5dff7019113a4323ae78b11f6b547b53]) successfully deleted
I20251023 20:15:19.840709 24094 raft_consensus.cc:493] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.841818 24094 raft_consensus.cc:515] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.843053 24094 leader_election.cc:290] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.843091 23144 master_service.cc:954] ReplaceTablet: received request to replace tablet 7645c36c983f438f8b1223256fd3ad2c from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:19.843111 23785 raft_consensus.cc:2243] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.843451 23942 raft_consensus.cc:697] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:19.843618 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0f8d8184a68453abd2792ccc7740aa1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:19.843675 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e0f8d8184a68453abd2792ccc7740aa1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:19.843924 23942 consensus_queue.cc:237] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:19.844169 23605 leader_election.cc:343] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:19.844331 23604 leader_election.cc:343] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.844450 23785 raft_consensus.cc:2272] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.844468 23604 leader_election.cc:304] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.847105 24094 raft_consensus.cc:2749] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:19.847844 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb915d114fc0421999d079c23cad8eb5 already in progress: creating tablet
I20251023 20:15:19.848244 23783 tablet_replica.cc:333] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.848505 23783 raft_consensus.cc:2243] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.848742 23783 raft_consensus.cc:2272] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.851477 23787 tablet_bootstrap.cc:492] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.852036 23787 ts_tablet_manager.cc:1403] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.001s
I20251023 20:15:19.852412 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bcdec7c16c943079e20bb486aecb431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.853351 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8e0012945b9e401793b99da25bc5d76c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.853685 23787 raft_consensus.cc:359] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.853875 23787 raft_consensus.cc:385] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:19.853919 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e0012945b9e401793b99da25bc5d76c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.853999 23787 raft_consensus.cc:740] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.854211 23787 consensus_queue.cc:260] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.854677 23787 ts_tablet_manager.cc:1434] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:19.854986 23787 tablet_bootstrap.cc:492] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.856667 23787 tablet_bootstrap.cc:654] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.857267 23783 ts_tablet_manager.cc:1916] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.858285 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_db52858e57b64c15a85a784ef9bff9b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:19.858496 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:19.858785 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db52858e57b64c15a85a784ef9bff9b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:19.860231 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19c58a406d564b048fb16a85aa0b754b consensus state has no leader
I20251023 20:15:19.860622 23786 tablet_bootstrap.cc:492] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:19.861586 23785 ts_tablet_manager.cc:1916] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:19.864804 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:19.865392 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_96e1782778fe489e8919612f34dc710f" } new_table_name: "default.Table_ababfc027cb24714ab70240be6fae889" modify_external_catalogs: true
I20251023 20:15:19.871651 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet c9bfbf0acdc84f2f8fe553694ee9897c
I20251023 20:15:19.872750 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:19.873008 23786 tablet_bootstrap.cc:654] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.873160 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.874475 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:19.878584 23784 ts_tablet_manager.cc:1929] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.878804 23784 log.cc:1199] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/487d9ee765d14aa2aa5668dc48fffe20
I20251023 20:15:19.883579 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4dbe3b1770374c18821af919d529fc09" } modify_external_catalogs: true
I20251023 20:15:19.883755 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4dbe3b1770374c18821af919d529fc09" } modify_external_catalogs: true
I20251023 20:15:19.884337 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_736b5fe2d49043f896fa85aeed517265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.884867 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_736b5fe2d49043f896fa85aeed517265 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:19.887377 24094 raft_consensus.cc:493] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.887619 24094 raft_consensus.cc:515] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.889776 23790 tablet_bootstrap.cc:492] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:19.890029 23790 ts_tablet_manager.cc:1403] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.140s user 0.008s sys 0.005s
I20251023 20:15:19.891144 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83da09ee0d10435ba89a19dc378d34eb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:19.894380 23605 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.896823 24094 leader_election.cc:290] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.908330 23783 ts_tablet_manager.cc:1929] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:19.908589 23783 log.cc:1199] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/451955aee3aa4bcb82252bcefbbcaefe
I20251023 20:15:19.909358 23783 ts_tablet_manager.cc:1950] T 451955aee3aa4bcb82252bcefbbcaefe P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:19.910030 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04f7dffd94f3442ea20045a512d7bd65 consensus state has no leader
I20251023 20:15:19.910913 23787 tablet_bootstrap.cc:492] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:19.911123 23787 ts_tablet_manager.cc:1403] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20251023 20:15:19.911959 23787 raft_consensus.cc:359] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.912286 23787 raft_consensus.cc:385] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.912410 23787 raft_consensus.cc:740] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.912691 23787 consensus_queue.cc:260] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.917148 23787 ts_tablet_manager.cc:1434] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251023 20:15:19.921257 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a6d0f3fe1c734fac92db5f6868a805de" } new_table_name: "default.Table_cf2a1b576c024ec89df2ce4b0ca48a29" modify_external_catalogs: true
I20251023 20:15:19.922868 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet 7f766bfe7dd94352abcaf34a200d00bc from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:19.927690 23787 tablet_bootstrap.cc:492] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:19.929629 23787 tablet_bootstrap.cc:654] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.929852 23790 raft_consensus.cc:359] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.930192 23790 raft_consensus.cc:385] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.930325 23790 raft_consensus.cc:740] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.930334 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bd447f978d1d4b8bad839b915907db66" } modify_external_catalogs: true
I20251023 20:15:19.930534 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bd447f978d1d4b8bad839b915907db66" } modify_external_catalogs: true
I20251023 20:15:19.930630 23790 consensus_queue.cc:260] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.933507 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 451955aee3aa4bcb82252bcefbbcaefe (table default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
I20251023 20:15:19.935106 23783 tablet_replica.cc:333] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:19.935107 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_aba84ed1ff684cc690f31c5615ec1700" } modify_external_catalogs: true
I20251023 20:15:19.935449 23783 raft_consensus.cc:2243] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:19.935431 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_aba84ed1ff684cc690f31c5615ec1700" } modify_external_catalogs: true
I20251023 20:15:19.935674 23783 raft_consensus.cc:2272] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:19.936174 23790 ts_tablet_manager.cc:1434] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20251023 20:15:19.936249 23783 ts_tablet_manager.cc:1916] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:19.938978 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb5a6d9a4509431f97b69a3b138b5829 consensus state has no leader
I20251023 20:15:19.895407 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83da09ee0d10435ba89a19dc378d34eb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:19.940308 23604 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:19.940454 23604 leader_election.cc:304] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:19.940956 24094 raft_consensus.cc:2749] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:19.941211 23784 ts_tablet_manager.cc:1950] T 487d9ee765d14aa2aa5668dc48fffe20 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:19.941569 23790 tablet_bootstrap.cc:492] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:19.942960 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7c03bf4f1b034b139999af6a349dccf0" } modify_external_catalogs: true
I20251023 20:15:19.943128 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7c03bf4f1b034b139999af6a349dccf0" } modify_external_catalogs: true
I20251023 20:15:19.944396 23790 tablet_bootstrap.cc:654] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:19.947042 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 487d9ee765d14aa2aa5668dc48fffe20 (table default.Table_561d6afe24b246c9909547a248a5fe7f [id=a0f9058297034e739a638f1747ddacfd]) successfully deleted
W20251023 20:15:19.948562 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7540354f8577482785e6341fa9022315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7540354f8577482785e6341fa9022315 already in progress: creating tablet
I20251023 20:15:19.945394 23786 tablet_bootstrap.cc:492] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:19.949326 23139 catalog_manager.cc:5649] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:19.949777 23139 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:19.950573 23786 ts_tablet_manager.cc:1403] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20251023 20:15:19.951582 23786 raft_consensus.cc:359] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.951885 23786 raft_consensus.cc:385] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:19.952013 23786 raft_consensus.cc:740] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:19.952342 23786 consensus_queue.cc:260] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:19.953819 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 21a508e875c449c5831968562263f1bf (DEFAULT_TABLE table=default.Table_365837ad05624d75855461852362db1c [id=18a582219c6248adac2c8e7e2611808d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.953908 23637 tablet_service.cc:1505] Processing CreateTablet for tablet c3d238e071cb4d49b173febd90bc909b (DEFAULT_TABLE table=default.Table_d491cd3951344335be8170dcdcb94c4c [id=2c61abbba33e4ed4805127246e3cd0ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.954300 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a508e875c449c5831968562263f1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.955018 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d238e071cb4d49b173febd90bc909b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.955282 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 21a508e875c449c5831968562263f1bf (DEFAULT_TABLE table=default.Table_365837ad05624d75855461852362db1c [id=18a582219c6248adac2c8e7e2611808d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.955636 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a508e875c449c5831968562263f1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.957429 23649 tablet_service.cc:1505] Processing CreateTablet for tablet efad9b4a8e2e446f917df7ca26fbf789 (DEFAULT_TABLE table=default.Table_a6f1c53dfe964d538a77e7b6570d9a9a [id=f2b154b7183f4ad0b4fc22a9d5f9620f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.957780 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efad9b4a8e2e446f917df7ca26fbf789. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.959769 23505 tablet_service.cc:1505] Processing CreateTablet for tablet efad9b4a8e2e446f917df7ca26fbf789 (DEFAULT_TABLE table=default.Table_a6f1c53dfe964d538a77e7b6570d9a9a [id=f2b154b7183f4ad0b4fc22a9d5f9620f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.960258 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efad9b4a8e2e446f917df7ca26fbf789. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.960367 23516 tablet_service.cc:1505] Processing CreateTablet for tablet c3d238e071cb4d49b173febd90bc909b (DEFAULT_TABLE table=default.Table_d491cd3951344335be8170dcdcb94c4c [id=2c61abbba33e4ed4805127246e3cd0ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.960754 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d238e071cb4d49b173febd90bc909b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.962319 23641 tablet_service.cc:1505] Processing CreateTablet for tablet e5af107f9a144edb9080b7135ab8aeb9 (DEFAULT_TABLE table=default.Table_86168bc39ef14e358f5356df1fe2e346 [id=89fe830110bb49bd891eb42bac7215d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.962543 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 94cd8880a78c4b4eb0c19ba5944b12f2 (DEFAULT_TABLE table=default.Table_240b16ca6e1543f2aab607c1d6a447fd [id=4aa2d573bbb34a3ebda2ac6376a387ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.962607 23639 tablet_service.cc:1505] Processing CreateTablet for tablet f03df77c344b4736a30725e4d9f72a35 (DEFAULT_TABLE table=default.Table_7908a191ee99400093d9f2deb05e52d2 [id=2ad9425327fb485d89c61e102c983b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.962728 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5af107f9a144edb9080b7135ab8aeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.962896 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94cd8880a78c4b4eb0c19ba5944b12f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.963119 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 92c50c4845524163834861438291a3ec (DEFAULT_TABLE table=default.Table_b01089ea89ef4536a351098a4caa38f0 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.963440 23647 tablet_service.cc:1505] Processing CreateTablet for tablet 9b766ff2dff34a8f80f70591dba54017 (DEFAULT_TABLE table=default.Table_7d29445c967d4d7fb9d42c9d178c677b [id=32d1b2b4b0c84ec48a98220a4def4664]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.963464 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c50c4845524163834861438291a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.964200 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03df77c344b4736a30725e4d9f72a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.964371 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 94cd8880a78c4b4eb0c19ba5944b12f2 (DEFAULT_TABLE table=default.Table_240b16ca6e1543f2aab607c1d6a447fd [id=4aa2d573bbb34a3ebda2ac6376a387ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.964643 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b766ff2dff34a8f80f70591dba54017. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.964735 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94cd8880a78c4b4eb0c19ba5944b12f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.967610 23497 tablet_service.cc:1505] Processing CreateTablet for tablet e5af107f9a144edb9080b7135ab8aeb9 (DEFAULT_TABLE table=default.Table_86168bc39ef14e358f5356df1fe2e346 [id=89fe830110bb49bd891eb42bac7215d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.967974 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5af107f9a144edb9080b7135ab8aeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.968809 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e95151c6b95f40328bc00ecbfcc67d1a" } new_table_name: "default.Table_b33239cad93e4f27aee7c9dbe7fab6d9" modify_external_catalogs: true
I20251023 20:15:19.969125 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 60348d0c00c14247860b9df14ddb82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:19.971663 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 6480cdc1afda4671a346044f232b083e (DEFAULT_TABLE table=default.Table_031b7f15cd9c4b17acd7c1c9c9102fbd [id=8b74dbf9b511448bab0acfe311fbb0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.972054 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6480cdc1afda4671a346044f232b083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.973387 23506 tablet_service.cc:1505] Processing CreateTablet for tablet 9b766ff2dff34a8f80f70591dba54017 (DEFAULT_TABLE table=default.Table_7d29445c967d4d7fb9d42c9d178c677b [id=32d1b2b4b0c84ec48a98220a4def4664]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.973742 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b766ff2dff34a8f80f70591dba54017. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:19.976430 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fea0afc8f63497d8b5cdf2352a20267 consensus state has no leader
I20251023 20:15:19.977488 23514 tablet_service.cc:1505] Processing CreateTablet for tablet f03df77c344b4736a30725e4d9f72a35 (DEFAULT_TABLE table=default.Table_7908a191ee99400093d9f2deb05e52d2 [id=2ad9425327fb485d89c61e102c983b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.977912 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03df77c344b4736a30725e4d9f72a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.980700 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 6480cdc1afda4671a346044f232b083e (DEFAULT_TABLE table=default.Table_031b7f15cd9c4b17acd7c1c9c9102fbd [id=8b74dbf9b511448bab0acfe311fbb0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.981108 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6480cdc1afda4671a346044f232b083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.981117 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c (DEFAULT_TABLE table=default.Table_6179bdbf369e4b32be88ca151e71e50f [id=ee282decb79a4f27908c253cf524f971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.981340 23509 tablet_service.cc:1505] Processing CreateTablet for tablet 92c50c4845524163834861438291a3ec (DEFAULT_TABLE table=default.Table_b01089ea89ef4536a351098a4caa38f0 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.981938 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377de4ea0bd6489284fac6b0fab52c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.982947 23147 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 3650822dd30c4c8e97982636ac692eae
I20251023 20:15:19.983120 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c50c4845524163834861438291a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.984877 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c (DEFAULT_TABLE table=default.Table_6179bdbf369e4b32be88ca151e71e50f [id=ee282decb79a4f27908c253cf524f971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:19.985363 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377de4ea0bd6489284fac6b0fab52c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:19.987659 24098 raft_consensus.cc:493] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:19.987838 24098 raft_consensus.cc:515] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:19.988615 24098 leader_election.cc:290] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:19.996739 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d82a467c784f46dab749a2bfaa0aedbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:19.997382 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d82a467c784f46dab749a2bfaa0aedbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.001389 23647 tablet_service.cc:1505] Processing CreateTablet for tablet 67e0701159c548c49e9e245820c45028 (DEFAULT_TABLE table=default.Table_7f5474847aa74113aca18fe96357be62 [id=ac08261334284759a3997df44867cc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.001950 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e0701159c548c49e9e245820c45028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.002233 24094 raft_consensus.cc:493] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.002384 24094 raft_consensus.cc:515] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.004125 24094 leader_election.cc:290] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.000870 23786 ts_tablet_manager.cc:1434] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20251023 20:15:20.004590 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6558265ff9e249c68123577958f1401a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:20.005710 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6558265ff9e249c68123577958f1401a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.006913 23787 tablet_bootstrap.cc:492] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.007102 23787 ts_tablet_manager.cc:1403] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20251023 20:15:20.007459 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 67e0701159c548c49e9e245820c45028 (DEFAULT_TABLE table=default.Table_7f5474847aa74113aca18fe96357be62 [id=ac08261334284759a3997df44867cc31]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:20.007916 23605 leader_election.cc:343] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:20.008116 23604 leader_election.cc:343] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.008611 23514 tablet_service.cc:1505] Processing CreateTablet for tablet a2d4ba34cc444baf9fe725bc81478517 (DEFAULT_TABLE table=default.Table_87caa0c69120413a92c6b65337baad14 [id=3227f5fd7d78482c99fb9f654ba8e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.008654 23604 leader_election.cc:304] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.009003 24094 raft_consensus.cc:2749] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.010790 24094 raft_consensus.cc:493] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.011142 24094 raft_consensus.cc:515] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.011891 23641 tablet_service.cc:1505] Processing CreateTablet for tablet a2d4ba34cc444baf9fe725bc81478517 (DEFAULT_TABLE table=default.Table_87caa0c69120413a92c6b65337baad14 [id=3227f5fd7d78482c99fb9f654ba8e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.012184 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 559a70cf17ad4145acdb3033e01ad217 (DEFAULT_TABLE table=default.Table_b98d2ec04db54e72a067731a38d21a56 [id=91a585d91b924838996b49fe349985f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.012317 23785 ts_tablet_manager.cc:1929] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.012478 23785 log.cc:1199] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/2e338b982a564ea5b1b15c2146a96fa2
I20251023 20:15:20.012931 23785 ts_tablet_manager.cc:1950] T 2e338b982a564ea5b1b15c2146a96fa2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:20.014230 23785 tablet_replica.cc:333] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:20.014561 23785 raft_consensus.cc:2243] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:20.014680 24094 leader_election.cc:290] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.014909 23787 raft_consensus.cc:359] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.015105 23785 pending_rounds.cc:70] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:20.015157 23787 raft_consensus.cc:385] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.015343 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f766bfe7dd94352abcaf34a200d00bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.015727 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 72d1dde8b0ac4812869ef233c18e550f (DEFAULT_TABLE table=default.Table_efda7d26e2124ec4b1eeca32fe1b545e [id=6f9ad1c9f64a44a4b856013bb8fd4bd1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:20.015941 23604 leader_election.cc:343] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.016582 23786 tablet_bootstrap.cc:492] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:20.016567 23787 raft_consensus.cc:740] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.018590 23784 tablet_replica.cc:333] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:20.019688 23784 raft_consensus.cc:2243] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.020139 23784 raft_consensus.cc:2272] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.022104 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b9e2bcb23af94936a9fbfd0817d9a51e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.022704 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9e2bcb23af94936a9fbfd0817d9a51e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.026286 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 2e338b982a564ea5b1b15c2146a96fa2 (table default.Table_4f134fc02d1c4a53a726595803ef505c [id=ff3dd4d1748144c492b64ccde7297729]) successfully deleted
I20251023 20:15:20.026793 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c95db7b5b9fd46acae8ffa7119915907" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.027669 24094 raft_consensus.cc:493] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.017300 23787 consensus_queue.cc:260] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.017484 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d4ba34cc444baf9fe725bc81478517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.016209 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 72d1dde8b0ac4812869ef233c18e550f (DEFAULT_TABLE table=default.Table_efda7d26e2124ec4b1eeca32fe1b545e [id=6f9ad1c9f64a44a4b856013bb8fd4bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.036898 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 46f700ea3db04a4bad539c943e850a44
I20251023 20:15:20.038796 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_2f964a2f8a294783802fdf58c31376c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.039343 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f964a2f8a294783802fdf58c31376c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.042346 23783 ts_tablet_manager.cc:1929] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.042551 24094 raft_consensus.cc:515] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.043285 24094 leader_election.cc:290] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.047031 23783 log.cc:1199] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e31aebfc87734ffeb2c0b25c07cb3f70
I20251023 20:15:20.046957 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 559a70cf17ad4145acdb3033e01ad217 (DEFAULT_TABLE table=default.Table_b98d2ec04db54e72a067731a38d21a56 [id=91a585d91b924838996b49fe349985f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.048156 23783 ts_tablet_manager.cc:1950] T e31aebfc87734ffeb2c0b25c07cb3f70 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:20.048280 23787 ts_tablet_manager.cc:1434] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20251023 20:15:20.048960 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.051659 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 48436f3d8f1443db990a2d10b668b70a
I20251023 20:15:20.052449 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.017591 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f766bfe7dd94352abcaf34a200d00bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:20.054680 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_273b73dbda4e41aea636fdc95149120a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.054919 23398 raft_consensus.cc:2468] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:20.018285 23373 tablet_service.cc:1505] Processing CreateTablet for tablet e5af107f9a144edb9080b7135ab8aeb9 (DEFAULT_TABLE table=default.Table_86168bc39ef14e358f5356df1fe2e346 [id=89fe830110bb49bd891eb42bac7215d7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:20.055281 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_273b73dbda4e41aea636fdc95149120a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.055500 23787 tablet_bootstrap.cc:492] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.055616 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5af107f9a144edb9080b7135ab8aeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.056854 23785 pending_rounds.cc:77] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082126527279104 op_type: NO_OP noop_request { }
I20251023 20:15:20.058128 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 336) took 1049 ms. Trace:
I20251023 20:15:20.058286 23783 rpcz_store.cc:276] 1023 20:15:19.008162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.008211 (+ 49us) service_pool.cc:225] Handling call
1023 20:15:19.935029 (+926818us) ts_tablet_manager.cc:1151] Deleting tablet e31aebfc87734ffeb2c0b25c07cb3f70
1023 20:15:19.935040 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:19.994184 (+ 59144us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.042202 (+ 48018us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.058036 (+ 15834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926534}
I20251023 20:15:20.059434 23783 tablet_replica.cc:333] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.059657 23783 raft_consensus.cc:2243] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:20.060115 23783 pending_rounds.cc:70] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Trying to abort 1 pending ops.
I20251023 20:15:20.060322 23783 pending_rounds.cc:77] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082129283874816 op_type: NO_OP noop_request { }
I20251023 20:15:20.060505 23783 raft_consensus.cc:2889] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:20.060629 23783 raft_consensus.cc:2272] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.018427 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 21a508e875c449c5831968562263f1bf (DEFAULT_TABLE table=default.Table_365837ad05624d75855461852362db1c [id=18a582219c6248adac2c8e7e2611808d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.061549 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a508e875c449c5831968562263f1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.062273 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 72408899f710470baaf6fdb268cb04c8
I20251023 20:15:20.062788 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet e31aebfc87734ffeb2c0b25c07cb3f70 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
I20251023 20:15:20.063647 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.018813 23365 tablet_service.cc:1505] Processing CreateTablet for tablet efad9b4a8e2e446f917df7ca26fbf789 (DEFAULT_TABLE table=default.Table_a6f1c53dfe964d538a77e7b6570d9a9a [id=f2b154b7183f4ad0b4fc22a9d5f9620f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.019212 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 94cd8880a78c4b4eb0c19ba5944b12f2 (DEFAULT_TABLE table=default.Table_240b16ca6e1543f2aab607c1d6a447fd [id=4aa2d573bbb34a3ebda2ac6376a387ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.065426 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efad9b4a8e2e446f917df7ca26fbf789. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.065729 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94cd8880a78c4b4eb0c19ba5944b12f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.063946 23785 raft_consensus.cc:2889] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:20.068027 23785 raft_consensus.cc:2272] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.068392 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72d1dde8b0ac4812869ef233c18e550f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.019546 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 9b766ff2dff34a8f80f70591dba54017 (DEFAULT_TABLE table=default.Table_7d29445c967d4d7fb9d42c9d178c677b [id=32d1b2b4b0c84ec48a98220a4def4664]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.020167 23374 tablet_service.cc:1505] Processing CreateTablet for tablet f03df77c344b4736a30725e4d9f72a35 (DEFAULT_TABLE table=default.Table_7908a191ee99400093d9f2deb05e52d2 [id=2ad9425327fb485d89c61e102c983b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.020678 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 92c50c4845524163834861438291a3ec (DEFAULT_TABLE table=default.Table_b01089ea89ef4536a351098a4caa38f0 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.021234 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 6480cdc1afda4671a346044f232b083e (DEFAULT_TABLE table=default.Table_031b7f15cd9c4b17acd7c1c9c9102fbd [id=8b74dbf9b511448bab0acfe311fbb0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.021750 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c (DEFAULT_TABLE table=default.Table_6179bdbf369e4b32be88ca151e71e50f [id=ee282decb79a4f27908c253cf524f971]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.024284 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 67e0701159c548c49e9e245820c45028 (DEFAULT_TABLE table=default.Table_7f5474847aa74113aca18fe96357be62 [id=ac08261334284759a3997df44867cc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.024786 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 72d1dde8b0ac4812869ef233c18e550f (DEFAULT_TABLE table=default.Table_efda7d26e2124ec4b1eeca32fe1b545e [id=6f9ad1c9f64a44a4b856013bb8fd4bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.025377 23370 tablet_service.cc:1505] Processing CreateTablet for tablet a2d4ba34cc444baf9fe725bc81478517 (DEFAULT_TABLE table=default.Table_87caa0c69120413a92c6b65337baad14 [id=3227f5fd7d78482c99fb9f654ba8e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.071465 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b766ff2dff34a8f80f70591dba54017. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.072552 23605 leader_election.cc:304] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.025914 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 559a70cf17ad4145acdb3033e01ad217 (DEFAULT_TABLE table=default.Table_b98d2ec04db54e72a067731a38d21a56 [id=91a585d91b924838996b49fe349985f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.073275 24094 raft_consensus.cc:2804] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:20.073396 24094 raft_consensus.cc:493] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:20.073514 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559a70cf17ad4145acdb3033e01ad217. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.073550 24094 raft_consensus.cc:3060] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:20.026886 23333 leader_election.cc:343] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: c95db7b5b9fd46acae8ffa7119915907
I20251023 20:15:20.027437 23368 tablet_service.cc:1505] Processing CreateTablet for tablet c3d238e071cb4d49b173febd90bc909b (DEFAULT_TABLE table=default.Table_d491cd3951344335be8170dcdcb94c4c [id=2c61abbba33e4ed4805127246e3cd0ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.046242 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e19f64a9ac44efba92928d0fa16819a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:20.074761 23605 leader_election.cc:343] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.076380 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7d29445c967d4d7fb9d42c9d178c677b" } new_table_name: "default.Table_4e035ee3714e4e47b7748fa23656760a" modify_external_catalogs: true
I20251023 20:15:20.078754 24094 raft_consensus.cc:515] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.082330 23787 tablet_bootstrap.cc:654] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.082861 23783 ts_tablet_manager.cc:1916] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.079342 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e0701159c548c49e9e245820c45028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.084834 23942 raft_consensus.cc:493] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.086572 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 7645c36c983f438f8b1223256fd3ad2c
I20251023 20:15:20.086627 23942 raft_consensus.cc:515] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.087086 23144 catalog_manager.cc:6719] ReplaceTablet: tablet 7645c36c983f438f8b1223256fd3ad2c deleted and replaced by tablet 82ae1a94b011412c851cd96946744fc4
I20251023 20:15:20.087369 23942 leader_election.cc:290] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.087725 24094 leader_election.cc:290] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.088198 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.089882 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_031b7f15cd9c4b17acd7c1c9c9102fbd" } modify_external_catalogs: true
I20251023 20:15:20.090085 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_031b7f15cd9c4b17acd7c1c9c9102fbd" } modify_external_catalogs: true
I20251023 20:15:20.096742 23787 tablet_bootstrap.cc:492] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.097306 23787 ts_tablet_manager.cc:1403] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20251023 20:15:20.098353 23787 raft_consensus.cc:359] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.098740 23787 raft_consensus.cc:385] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.098906 23787 raft_consensus.cc:740] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.099349 23787 consensus_queue.cc:260] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.099651 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 7f766bfe7dd94352abcaf34a200d00bc
I20251023 20:15:20.099952 23787 ts_tablet_manager.cc:1434] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:20.100116 23142 catalog_manager.cc:6719] ReplaceTablet: tablet 7f766bfe7dd94352abcaf34a200d00bc deleted and replaced by tablet 53f33020319d48c791e1156c67b55be6
I20251023 20:15:20.100368 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet ac55e14745534a02a2f6a422cab6b149 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.101043 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f766bfe7dd94352abcaf34a200d00bc with delete_type TABLET_DATA_DELETED (replaced by tablet 53f33020319d48c791e1156c67b55be6) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.104327 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6480cdc1afda4671a346044f232b083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.105914 23787 tablet_bootstrap.cc:492] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.108807 23787 tablet_bootstrap.cc:654] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:20.109688 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 47299.8us
I20251023 20:15:20.110551 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e19f64a9ac44efba92928d0fa16819a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.111588 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c50c4845524163834861438291a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:20.111668 23604 leader_election.cc:343] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.111825 23604 leader_election.cc:304] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.112279 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.112578 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f766bfe7dd94352abcaf34a200d00bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:20.113288 23604 leader_election.cc:343] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.113095 23379 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.113205 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.113469 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:20.113535 23604 leader_election.cc:343] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.112215 23785 ts_tablet_manager.cc:1916] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.114773 23786 tablet_bootstrap.cc:654] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.114897 24094 raft_consensus.cc:2749] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.115070 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f766bfe7dd94352abcaf34a200d00bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:20.115283 23399 raft_consensus.cc:3060] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:20.115319 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d4ba34cc444baf9fe725bc81478517. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.117645 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f766bfe7dd94352abcaf34a200d00bc with delete_type TABLET_DATA_DELETED (replaced by tablet 53f33020319d48c791e1156c67b55be6) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.118822 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d238e071cb4d49b173febd90bc909b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.120244 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.121948 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f766bfe7dd94352abcaf34a200d00bc with delete_type TABLET_DATA_DELETED (replaced by tablet 53f33020319d48c791e1156c67b55be6) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.122251 23373 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.122540 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72d1dde8b0ac4812869ef233c18e550f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:20.122848 23333 leader_election.cc:343] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.123004 23333 leader_election.cc:304] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.123286 24098 raft_consensus.cc:2749] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.124452 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:20.125236 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559a70cf17ad4145acdb3033e01ad217. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.126310 23399 raft_consensus.cc:2468] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:20.126441 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d4ba34cc444baf9fe725bc81478517. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:20.127170 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 244fa327fad74aad9e57a7fbcbf908bb consensus state has no leader
W20251023 20:15:20.128535 23605 leader_election.cc:343] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.128813 23605 leader_election.cc:304] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.129925 23605 leader_election.cc:304] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.130273 23942 raft_consensus.cc:2749] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.130296 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.130492 23942 raft_consensus.cc:2804] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:20.130661 23942 raft_consensus.cc:697] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:20.130795 23509 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.130920 23942 consensus_queue.cc:237] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.131290 23716 heartbeater.cc:344] Connected to a master server at 127.22.127.190:34331
I20251023 20:15:20.131417 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.136092 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377de4ea0bd6489284fac6b0fab52c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.136395 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e0701159c548c49e9e245820c45028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.137112 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_464dc629d50b4995b30ef5c463e9f316"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.137778 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_464dc629d50b4995b30ef5c463e9f316 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.138640 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 559a70cf17ad4145acdb3033e01ad217. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.142381 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72d1dde8b0ac4812869ef233c18e550f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.147984 24094 raft_consensus.cc:493] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.148221 24094 raft_consensus.cc:515] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.149534 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3894dd4feeea4c16861a6c9520059474" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:20.150225 23604 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.150491 24094 leader_election.cc:290] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.107992 23784 ts_tablet_manager.cc:1916] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.158799 23787 tablet_bootstrap.cc:492] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.159017 23787 ts_tablet_manager.cc:1403] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.002s
I20251023 20:15:20.160002 23787 raft_consensus.cc:359] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.160554 23790 tablet_bootstrap.cc:492] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:20.160459 23787 raft_consensus.cc:385] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.162474 23787 raft_consensus.cc:740] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.162811 23787 consensus_queue.cc:260] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.163466 23787 ts_tablet_manager.cc:1434] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20251023 20:15:20.163470 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_86c8c43350214e2eb31dc3b2e1f5333b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.163961 23787 tablet_bootstrap.cc:492] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:20.164105 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86c8c43350214e2eb31dc3b2e1f5333b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.165647 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_eb73fed1f9bb425daff2e3278ad80ece"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.166186 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb73fed1f9bb425daff2e3278ad80ece in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.166237 23787 tablet_bootstrap.cc:654] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.181349 23787 tablet_bootstrap.cc:492] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.181633 23787 ts_tablet_manager.cc:1403] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20251023 20:15:20.182643 23787 raft_consensus.cc:359] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.182978 23787 raft_consensus.cc:385] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.183120 23787 raft_consensus.cc:740] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.183485 23787 consensus_queue.cc:260] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.184167 23787 ts_tablet_manager.cc:1434] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:20.184527 23787 tablet_bootstrap.cc:492] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.186460 23787 tablet_bootstrap.cc:654] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.197268 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03df77c344b4736a30725e4d9f72a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.202227 23787 tablet_bootstrap.cc:492] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.202502 23787 ts_tablet_manager.cc:1403] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20251023 20:15:20.202886 23786 tablet_bootstrap.cc:492] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:20.203030 23786 ts_tablet_manager.cc:1403] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.198s user 0.003s sys 0.008s
I20251023 20:15:20.203619 23787 raft_consensus.cc:359] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.203896 23787 raft_consensus.cc:385] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.204108 23787 raft_consensus.cc:740] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.204443 23787 consensus_queue.cc:260] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.206033 23783 ts_tablet_manager.cc:1929] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:20.206179 23783 log.cc:1199] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/1bcdec7c16c943079e20bb486aecb431
I20251023 20:15:20.206149 23786 raft_consensus.cc:359] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.206348 23786 raft_consensus.cc:385] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.206466 23786 raft_consensus.cc:740] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.206722 23783 ts_tablet_manager.cc:1950] T 1bcdec7c16c943079e20bb486aecb431 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:20.206710 23786 consensus_queue.cc:260] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.206733 23787 ts_tablet_manager.cc:1434] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:20.207223 23787 tablet_bootstrap.cc:492] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.209407 23787 tablet_bootstrap.cc:654] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.211999 23786 ts_tablet_manager.cc:1434] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251023 20:15:20.214324 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_664e281e682d4426b64cb60b0af1ac18" } new_table_name: "default.Table_efbc2125b34c4ea48c56caa5c0b31909" modify_external_catalogs: true
I20251023 20:15:20.214829 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 62acc395fb294965a1ab476d37f9fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a27f192792cf41288d69ca15ab538472) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.215286 23942 raft_consensus.cc:493] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.215453 23942 raft_consensus.cc:515] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.216082 23942 leader_election.cc:290] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.219368 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fc7e18054c440098dc2470143ea7fae" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:20.220009 23604 leader_election.cc:343] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.223157 23787 tablet_bootstrap.cc:492] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.223347 23787 ts_tablet_manager.cc:1403] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
W20251023 20:15:20.223620 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b766ff2dff34a8f80f70591dba54017 consensus state has no leader
I20251023 20:15:20.224212 23787 raft_consensus.cc:359] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.224426 23787 raft_consensus.cc:385] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.224506 23787 raft_consensus.cc:740] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.225184 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f21169f4f8814e739bc80b279a060a39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.225368 23786 tablet_bootstrap.cc:492] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:20.225807 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f21169f4f8814e739bc80b279a060a39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.226986 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet ac55e14745534a02a2f6a422cab6b149
I20251023 20:15:20.227499 23142 catalog_manager.cc:6719] ReplaceTablet: tablet ac55e14745534a02a2f6a422cab6b149 deleted and replaced by tablet 473324b22da14618a84aa1136d081993
I20251023 20:15:20.227838 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_775428d3af964d538226c836574a899d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.228050 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.228214 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:20.228354 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_775428d3af964d538226c836574a899d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.229952 23787 consensus_queue.cc:260] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.230829 23787 ts_tablet_manager.cc:1434] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:20.231161 23787 tablet_bootstrap.cc:492] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.231324 23786 tablet_bootstrap.cc:654] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.233161 23787 tablet_bootstrap.cc:654] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.233650 23784 ts_tablet_manager.cc:1929] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.233912 23784 log.cc:1199] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/42ac1c6b30e048c58cfb7fc6ad0b5e69
I20251023 20:15:20.234473 23784 ts_tablet_manager.cc:1950] T 42ac1c6b30e048c58cfb7fc6ad0b5e69 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:20.235391 23787 tablet_bootstrap.cc:492] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.235517 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 42ac1c6b30e048c58cfb7fc6ad0b5e69 (table default.Table_3320634019ac4bf48649eff33160d808 [id=c06c5925ec4e418e91167cd090df7f81]) successfully deleted
I20251023 20:15:20.235663 23787 ts_tablet_manager.cc:1403] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251023 20:15:20.236642 23787 raft_consensus.cc:359] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.236922 23787 raft_consensus.cc:385] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.237123 23787 raft_consensus.cc:740] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:20.237183 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
I20251023 20:15:20.237546 23784 tablet_replica.cc:333] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:20.237473 23787 consensus_queue.cc:260] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.237918 23784 raft_consensus.cc:2243] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.238116 23787 ts_tablet_manager.cc:1434] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:20.238301 23784 raft_consensus.cc:2272] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.238472 23787 tablet_bootstrap.cc:492] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.239411 23784 ts_tablet_manager.cc:1916] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.240991 23787 tablet_bootstrap.cc:654] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.241961 23786 tablet_bootstrap.cc:492] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:20.242143 23786 ts_tablet_manager.cc:1403] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251023 20:15:20.243088 23786 raft_consensus.cc:359] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.243377 23786 raft_consensus.cc:385] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.243469 23786 raft_consensus.cc:740] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.246568 23787 tablet_bootstrap.cc:492] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.247825 23786 consensus_queue.cc:260] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.249025 23787 ts_tablet_manager.cc:1403] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20251023 20:15:20.249936 23787 raft_consensus.cc:359] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.250202 23787 raft_consensus.cc:385] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.250325 23787 raft_consensus.cc:740] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.250605 23787 consensus_queue.cc:260] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.251170 23787 ts_tablet_manager.cc:1434] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:20.251521 23787 tablet_bootstrap.cc:492] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.252708 23790 ts_tablet_manager.cc:1403] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.219s user 0.003s sys 0.002s
I20251023 20:15:20.253412 23786 ts_tablet_manager.cc:1434] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20251023 20:15:20.255963 23786 tablet_bootstrap.cc:492] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:20.256279 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3894dd4feeea4c16861a6c9520059474" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:20.257172 23373 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:20.257587 23605 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.257776 23605 leader_election.cc:304] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.258731 23942 raft_consensus.cc:2749] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.259946 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 337) took 1176 ms. Trace:
I20251023 20:15:20.260077 23783 rpcz_store.cc:276] 1023 20:15:19.083483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.083550 (+ 67us) service_pool.cc:225] Handling call
1023 20:15:20.059317 (+975767us) ts_tablet_manager.cc:1151] Deleting tablet 1bcdec7c16c943079e20bb486aecb431
1023 20:15:20.059327 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.171261 (+111934us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.205913 (+ 34652us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.259745 (+ 53832us) spinlock_profiling.cc:243] Waited 52.5 ms on lock 0x616000102d54. stack: 00007fb96b1373e6 00007fb96b1371a0 00007fb96a11529d 0000000000983974 000000000098393d 000000000098330a 00007fb976fe0daa 00007fb976f9f174 00007fb978330336 00007fb97aeef204 00007fb97af21e54 00007fb97af028f0 00007fb97af0267d 0000000000a880ec 00007fb96b17d76f 00007fb96b180a3b
1023 20:15:20.259836 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120791424,"tablet-delete.queue_time_us":975478}
I20251023 20:15:20.260636 23783 tablet_replica.cc:333] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.260859 23783 raft_consensus.cc:2243] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.260967 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 1bcdec7c16c943079e20bb486aecb431 (table default.Table_6f178deaeeb94c4da6a40568c635e0b5 [id=cc451577eb8b4ba7937b8dbe4e859bc8]) successfully deleted
I20251023 20:15:20.261104 23783 raft_consensus.cc:2272] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.261693 23783 ts_tablet_manager.cc:1916] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.261911 23787 tablet_bootstrap.cc:654] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:20.264323 23604 consensus_peers.cc:597] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:20.265992 23786 tablet_bootstrap.cc:654] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.259400 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fc7e18054c440098dc2470143ea7fae" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:20.267796 23605 leader_election.cc:343] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.267941 23605 leader_election.cc:304] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.268371 23942 raft_consensus.cc:2749] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.269595 23790 raft_consensus.cc:359] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.270143 23790 raft_consensus.cc:385] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.270254 23790 raft_consensus.cc:740] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.290623 23790 consensus_queue.cc:260] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.293607 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_59bd96856079400e8eda1b518a1ec457" } new_table_name: "default.Table_3ef388667712400bbd0f19475a8ed714" modify_external_catalogs: true
I20251023 20:15:20.293915 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet caaccec5866a4e95afe9ccbf1f5bc494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.295404 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_322349f14bf94a7cbd9cb6cb92793905"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.296051 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_322349f14bf94a7cbd9cb6cb92793905 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.305452 23790 ts_tablet_manager.cc:1434] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20251023 20:15:20.305859 23790 tablet_bootstrap.cc:492] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:20.312314 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_07cfd6b64da94bbfa85cb6c533f550c5" } new_table_name: "default.Table_c9ccf1142a9b449f91e99360376a7b22" modify_external_catalogs: true
I20251023 20:15:20.319785 23942 raft_consensus.cc:493] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.320472 23942 raft_consensus.cc:515] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.321215 23942 leader_election.cc:290] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.322136 23783 ts_tablet_manager.cc:1929] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.322260 23783 log.cc:1199] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/6558265ff9e249c68123577958f1401a
I20251023 20:15:20.322291 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1229b9e0384a82811f244d2bf4c8ce" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.322304 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1229b9e0384a82811f244d2bf4c8ce" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:20.322613 23783 ts_tablet_manager.cc:1950] T 6558265ff9e249c68123577958f1401a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:20.322867 23604 leader_election.cc:343] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:20.323158 23605 leader_election.cc:343] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.323320 23605 leader_election.cc:304] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.323405 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 338) took 1239 ms. Trace:
I20251023 20:15:20.323551 23783 rpcz_store.cc:276] 1023 20:15:19.084266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.084310 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:20.260490 (+1176180us) ts_tablet_manager.cc:1151] Deleting tablet 6558265ff9e249c68123577958f1401a
1023 20:15:20.260500 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.278615 (+ 18115us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.322003 (+ 43388us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.323362 (+ 1359us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175815}
I20251023 20:15:20.323812 23942 raft_consensus.cc:2749] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.324167 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 6558265ff9e249c68123577958f1401a (table default.Table_cf2a1b576c024ec89df2ce4b0ca48a29 [id=991a6048a0894f9f95f52a8f0b881db6]) successfully deleted
I20251023 20:15:20.327152 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet 352812b56f8d4feb818a364d6adfd86d from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.327028 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:20.327374 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:20.327875 24105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5a310b4a514c0f812bda133c756062. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:20.328264 24105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:20.339978 23783 tablet_replica.cc:333] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.340238 23783 raft_consensus.cc:2243] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.344331 23783 raft_consensus.cc:2272] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.344280 23879 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae5a310b4a514c0f812bda133c756062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:20.344520 23879 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:20.351672 23790 tablet_bootstrap.cc:654] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:20.352777 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 473324b22da14618a84aa1136d081993 has no consensus state
I20251023 20:15:20.353245 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_79be15cb4350452b90277004ec803c8a" } modify_external_catalogs: true
I20251023 20:15:20.353467 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_79be15cb4350452b90277004ec803c8a" } modify_external_catalogs: true
I20251023 20:15:20.354249 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet 514cb3b1ed024041ac160a7d9acff618 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.355378 23942 raft_consensus.cc:493] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.355579 23942 raft_consensus.cc:515] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.356292 23942 leader_election.cc:290] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.344990 23783 ts_tablet_manager.cc:1916] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.357437 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e76f2e26dcd4cdfbf72de899f5a1458" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:20.357882 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e76f2e26dcd4cdfbf72de899f5a1458" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:20.358587 23605 leader_election.cc:343] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:20.359066 23604 leader_election.cc:343] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.359205 23604 leader_election.cc:304] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.359673 23942 raft_consensus.cc:2749] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.370514 23790 tablet_bootstrap.cc:492] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:20.370790 23790 ts_tablet_manager.cc:1403] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.001s
I20251023 20:15:20.371809 23790 raft_consensus.cc:359] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.371886 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0843993aca2340378776a12d7b3a5d4c" } modify_external_catalogs: true
I20251023 20:15:20.372100 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0843993aca2340378776a12d7b3a5d4c" } modify_external_catalogs: true
I20251023 20:15:20.372195 23790 raft_consensus.cc:385] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.372352 23790 raft_consensus.cc:740] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.372655 23790 consensus_queue.cc:260] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.373270 23790 ts_tablet_manager.cc:1434] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:20.373963 23787 tablet_bootstrap.cc:492] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.374130 23787 ts_tablet_manager.cc:1403] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.123s user 0.022s sys 0.057s
I20251023 20:15:20.375000 23787 raft_consensus.cc:359] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.375277 23787 raft_consensus.cc:385] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.375391 23787 raft_consensus.cc:740] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.375837 23790 tablet_bootstrap.cc:492] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:20.378101 23794 raft_consensus.cc:493] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.378291 23794 raft_consensus.cc:515] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.378952 23794 leader_election.cc:290] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:20.388135 23783 ts_tablet_manager.cc:1929] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.388302 23783 log.cc:1199] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/8e19f64a9ac44efba92928d0fa16819a
W20251023 20:15:20.382854 23471 leader_election.cc:343] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 62acc395fb294965a1ab476d37f9fab1
I20251023 20:15:20.388700 23783 ts_tablet_manager.cc:1950] T 8e19f64a9ac44efba92928d0fa16819a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:20.389349 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a7306245d7b41cfab795450761144f5 consensus state has no leader
W20251023 20:15:20.389380 23470 leader_election.cc:343] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 62acc395fb294965a1ab476d37f9fab1
I20251023 20:15:20.389632 23470 leader_election.cc:304] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:20.390101 23794 raft_consensus.cc:2749] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.390470 23787 consensus_queue.cc:260] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.391050 23143 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:20.391237 23787 ts_tablet_manager.cc:1434] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251023 20:15:20.391547 23787 tablet_bootstrap.cc:492] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.391525 23143 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_af01354e99f74972ac276e46643f9dba [id=84aabbb61e1c4d778b8e07ebdddbc477]). Expected version 1 got 0
I20251023 20:15:20.391769 23143 catalog_manager.cc:5649] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:20.393581 23787 tablet_bootstrap.cc:654] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.394719 23790 tablet_bootstrap.cc:654] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.395711 23145 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ae910ac41fd4e349f159ab27e44375a from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.400652 23497 tablet_service.cc:1505] Processing CreateTablet for tablet ca11ddd5d97544cf916b87fd3dff219d (DEFAULT_TABLE table=default.Table_db52858e57b64c15a85a784ef9bff9b4 [id=2c4fd35e5445474a96d35f714922aac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.401150 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca11ddd5d97544cf916b87fd3dff219d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.401604 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e (DEFAULT_TABLE table=default.Table_736b5fe2d49043f896fa85aeed517265 [id=9e1ee844cf9e41e29059059fb01fb5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.402032 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.402283 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 29c2c976d6a3440394f478a31abc7134 (DEFAULT_TABLE table=default.Table_43dc7ae468fd42cb867849c762736c66 [id=d53659f9a30541ae913e26824a9ae7d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.402629 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c2c976d6a3440394f478a31abc7134. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.405352 23509 tablet_service.cc:1505] Processing CreateTablet for tablet 882f7bf08e2f4b5382702c3d6eaf201e (DEFAULT_TABLE table=default.Table_39fdd0d19cae4e928fdee07e74900b4b [id=356410224c0f44bdacbaa543e9dee27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.405686 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882f7bf08e2f4b5382702c3d6eaf201e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.405643 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 882f7bf08e2f4b5382702c3d6eaf201e (DEFAULT_TABLE table=default.Table_39fdd0d19cae4e928fdee07e74900b4b [id=356410224c0f44bdacbaa543e9dee27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.406061 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882f7bf08e2f4b5382702c3d6eaf201e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.408800 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e (DEFAULT_TABLE table=default.Table_736b5fe2d49043f896fa85aeed517265 [id=9e1ee844cf9e41e29059059fb01fb5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.409351 23639 tablet_service.cc:1505] Processing CreateTablet for tablet ca11ddd5d97544cf916b87fd3dff219d (DEFAULT_TABLE table=default.Table_db52858e57b64c15a85a784ef9bff9b4 [id=2c4fd35e5445474a96d35f714922aac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.409502 23647 tablet_service.cc:1505] Processing CreateTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 (DEFAULT_TABLE table=default.Table_c37d72052930485b819533573f3710ac [id=d38b70b1f8e147e2aeadf0fce048c2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.409845 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 27b593b020b34a969f77d2f68178e89d (DEFAULT_TABLE table=default.Table_baf8abd9a132412a8b53ac8d9e044dcf [id=51ea9f032fdd40bb9f0ea42bb80c5cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.410209 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 27b593b020b34a969f77d2f68178e89d (DEFAULT_TABLE table=default.Table_baf8abd9a132412a8b53ac8d9e044dcf [id=51ea9f032fdd40bb9f0ea42bb80c5cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.410563 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 29c2c976d6a3440394f478a31abc7134 (DEFAULT_TABLE table=default.Table_43dc7ae468fd42cb867849c762736c66 [id=d53659f9a30541ae913e26824a9ae7d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.410651 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b593b020b34a969f77d2f68178e89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.402037 23514 tablet_service.cc:1505] Processing CreateTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 (DEFAULT_TABLE table=default.Table_c37d72052930485b819533573f3710ac [id=d38b70b1f8e147e2aeadf0fce048c2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.411540 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f9935f5e4f4daa9df8fe1611ed76c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.413408 24137 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251023 20:15:20.414115 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.417591 23398 raft_consensus.cc:1275] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Refusing update from remote peer 9bc6522db11e4960a9aad84435371a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:20.418251 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b593b020b34a969f77d2f68178e89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.419162 23784 ts_tablet_manager.cc:1929] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.419313 23784 log.cc:1199] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/60348d0c00c14247860b9df14ddb82b9
I20251023 20:15:20.419705 23784 ts_tablet_manager.cc:1950] T 60348d0c00c14247860b9df14ddb82b9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:20.420746 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca11ddd5d97544cf916b87fd3dff219d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.421356 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f9935f5e4f4daa9df8fe1611ed76c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.422428 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 60348d0c00c14247860b9df14ddb82b9 (table default.Table_baf231bb8eaa40eeb01169ddcf5161bd [id=94828b76eb8a4afdac4136fc626b6147]) successfully deleted
W20251023 20:15:20.422739 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 48436f3d8f1443db990a2d10b668b70a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48436f3d8f1443db990a2d10b668b70a already in progress: creating tablet
W20251023 20:15:20.422966 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7f766bfe7dd94352abcaf34a200d00bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f766bfe7dd94352abcaf34a200d00bc already in progress: creating tablet
W20251023 20:15:20.423174 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:20.423370 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7645c36c983f438f8b1223256fd3ad2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7645c36c983f438f8b1223256fd3ad2c already in progress: creating tablet
I20251023 20:15:20.424315 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c2c976d6a3440394f478a31abc7134. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.427763 24094 consensus_queue.cc:1048] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:20.433517 23786 tablet_bootstrap.cc:492] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:20.433676 23786 ts_tablet_manager.cc:1403] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.178s user 0.002s sys 0.002s
I20251023 20:15:20.437484 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 76380b341a304d6ba31648fbd1548f05 (DEFAULT_TABLE table=default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.439486 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 76380b341a304d6ba31648fbd1548f05 (DEFAULT_TABLE table=default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.439576 23511 tablet_service.cc:1505] Processing CreateTablet for tablet e18c1df87ee640148f5e827ee168fea6 (DEFAULT_TABLE table=default.Table_c760c475c8554c89aeb6fd77fd422797 [id=886ab40a90ab4410aa13fabf17a1ea91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.440039 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76380b341a304d6ba31648fbd1548f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.441030 23637 tablet_service.cc:1505] Processing CreateTablet for tablet e18c1df87ee640148f5e827ee168fea6 (DEFAULT_TABLE table=default.Table_c760c475c8554c89aeb6fd77fd422797 [id=886ab40a90ab4410aa13fabf17a1ea91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.441435 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18c1df87ee640148f5e827ee168fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.441486 23786 raft_consensus.cc:359] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.441704 23786 raft_consensus.cc:385] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.441831 23786 raft_consensus.cc:740] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.442138 23786 consensus_queue.cc:260] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.442775 23786 ts_tablet_manager.cc:1434] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:20.449806 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 76380b341a304d6ba31648fbd1548f05 (DEFAULT_TABLE table=default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.450354 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76380b341a304d6ba31648fbd1548f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.451604 23370 tablet_service.cc:1505] Processing CreateTablet for tablet e18c1df87ee640148f5e827ee168fea6 (DEFAULT_TABLE table=default.Table_c760c475c8554c89aeb6fd77fd422797 [id=886ab40a90ab4410aa13fabf17a1ea91]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.452040 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18c1df87ee640148f5e827ee168fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.453996 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76380b341a304d6ba31648fbd1548f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.455205 23361 tablet_service.cc:1505] Processing CreateTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 (DEFAULT_TABLE table=default.Table_c37d72052930485b819533573f3710ac [id=d38b70b1f8e147e2aeadf0fce048c2a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.455595 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1f9935f5e4f4daa9df8fe1611ed76c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.462764 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18c1df87ee640148f5e827ee168fea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.462802 23516 tablet_service.cc:1505] Processing CreateTablet for tablet e658aa360e524a9f9bdb35c1f68ae426 (DEFAULT_TABLE table=default.Table_25427c3d36fb4ba4a01ac362787c81b8 [id=248b68f25e64412ba27e4c74b27c59d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.463239 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e658aa360e524a9f9bdb35c1f68ae426. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.469337 23786 tablet_bootstrap.cc:492] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:20.470741 23996 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" instance_seqno: 1761250513786292) as {username='slave'} at 127.22.127.130:50915; Asking this server to re-register.
I20251023 20:15:20.472869 23373 tablet_service.cc:1505] Processing CreateTablet for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (DEFAULT_TABLE table=default.Table_a21f26a0638f42d682abb0448b2a53e5 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.460812 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 27b593b020b34a969f77d2f68178e89d (DEFAULT_TABLE table=default.Table_baf8abd9a132412a8b53ac8d9e044dcf [id=51ea9f032fdd40bb9f0ea42bb80c5cbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.461292 23376 tablet_service.cc:1505] Processing CreateTablet for tablet ca11ddd5d97544cf916b87fd3dff219d (DEFAULT_TABLE table=default.Table_db52858e57b64c15a85a784ef9bff9b4 [id=2c4fd35e5445474a96d35f714922aac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.461684 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 29c2c976d6a3440394f478a31abc7134 (DEFAULT_TABLE table=default.Table_43dc7ae468fd42cb867849c762736c66 [id=d53659f9a30541ae913e26824a9ae7d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.461907 23362 tablet_service.cc:1505] Processing CreateTablet for tablet e658aa360e524a9f9bdb35c1f68ae426 (DEFAULT_TABLE table=default.Table_25427c3d36fb4ba4a01ac362787c81b8 [id=248b68f25e64412ba27e4c74b27c59d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.462324 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e (DEFAULT_TABLE table=default.Table_736b5fe2d49043f896fa85aeed517265 [id=9e1ee844cf9e41e29059059fb01fb5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.466859 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 882f7bf08e2f4b5382702c3d6eaf201e (DEFAULT_TABLE table=default.Table_39fdd0d19cae4e928fdee07e74900b4b [id=356410224c0f44bdacbaa543e9dee27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.468086 23369 tablet_service.cc:1505] Processing CreateTablet for tablet e194d03a6aa946fbb2c79d91b196440b (DEFAULT_TABLE table=default.Table_b6185784138b4c4b938effd9e433c447 [id=75026c75215e419983355275cf38f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.473342 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.474381 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b593b020b34a969f77d2f68178e89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.475303 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882f7bf08e2f4b5382702c3d6eaf201e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.475665 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e194d03a6aa946fbb2c79d91b196440b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.477705 23786 tablet_bootstrap.cc:654] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.478118 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e658aa360e524a9f9bdb35c1f68ae426. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.479307 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca11ddd5d97544cf916b87fd3dff219d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.480487 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c2c976d6a3440394f478a31abc7134. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.482801 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.483433 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 339) took 1327 ms. Trace:
I20251023 20:15:20.483644 23783 rpcz_store.cc:276] 1023 20:15:19.122696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.122751 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:20.339871 (+1217120us) ts_tablet_manager.cc:1151] Deleting tablet 8e19f64a9ac44efba92928d0fa16819a
1023 20:15:20.339883 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.360931 (+ 21048us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.388016 (+ 27085us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.450133 (+ 62117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216865}
I20251023 20:15:20.484586 23582 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.484581 23514 tablet_service.cc:1505] Processing CreateTablet for tablet e194d03a6aa946fbb2c79d91b196440b (DEFAULT_TABLE table=default.Table_b6185784138b4c4b938effd9e433c447 [id=75026c75215e419983355275cf38f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.484788 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 8e19f64a9ac44efba92928d0fa16819a (table default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]) successfully deleted
W20251023 20:15:20.485257 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 46f700ea3db04a4bad539c943e850a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f700ea3db04a4bad539c943e850a44 already in progress: creating tablet
I20251023 20:15:20.486937 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e194d03a6aa946fbb2c79d91b196440b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.490235 23784 tablet_replica.cc:333] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:20.490339 23783 tablet_replica.cc:333] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.490609 23783 raft_consensus.cc:2243] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.493017 23365 tablet_service.cc:1505] Processing CreateTablet for tablet d62dbf044acd4459b26d2e7e50f6206e (DEFAULT_TABLE table=default.Table_8e0012945b9e401793b99da25bc5d76c [id=a0f3574408d0493e898036800ff93a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.493474 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d62dbf044acd4459b26d2e7e50f6206e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.502290 23790 tablet_bootstrap.cc:492] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:20.502584 23790 ts_tablet_manager.cc:1403] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.127s user 0.006s sys 0.003s
I20251023 20:15:20.503799 23790 raft_consensus.cc:359] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.504159 23790 raft_consensus.cc:385] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.504302 23790 raft_consensus.cc:740] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.504652 23790 consensus_queue.cc:260] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.505335 23790 ts_tablet_manager.cc:1434] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251023 20:15:20.505816 23790 tablet_bootstrap.cc:492] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:20.510892 23997 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9bc6522db11e4960a9aad84435371a32" instance_seqno: 1761250514256798) as {username='slave'} at 127.22.127.131:53181; Asking this server to re-register.
I20251023 20:15:20.512887 23784 raft_consensus.cc:2243] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.513566 23784 raft_consensus.cc:2272] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.513442 23509 tablet_service.cc:1505] Processing CreateTablet for tablet d62dbf044acd4459b26d2e7e50f6206e (DEFAULT_TABLE table=default.Table_8e0012945b9e401793b99da25bc5d76c [id=a0f3574408d0493e898036800ff93a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.514267 23511 tablet_service.cc:1505] Processing CreateTablet for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (DEFAULT_TABLE table=default.Table_a21f26a0638f42d682abb0448b2a53e5 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.517349 23790 tablet_bootstrap.cc:654] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.518127 23645 tablet_service.cc:1505] Processing CreateTablet for tablet e658aa360e524a9f9bdb35c1f68ae426 (DEFAULT_TABLE table=default.Table_25427c3d36fb4ba4a01ac362787c81b8 [id=248b68f25e64412ba27e4c74b27c59d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.518603 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e658aa360e524a9f9bdb35c1f68ae426. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.518935 23784 ts_tablet_manager.cc:1916] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.519151 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:20.520509 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0858358aabff47a69dc77f4669b225c5 consensus state has no leader
I20251023 20:15:20.522459 23997 ts_manager.cc:194] Registered new tserver with Master: ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.523375 23783 raft_consensus.cc:2272] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.523813 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d62dbf044acd4459b26d2e7e50f6206e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.524070 24094 raft_consensus.cc:493] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.524238 24094 raft_consensus.cc:515] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.524767 23783 ts_tablet_manager.cc:1916] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.527181 24094 leader_election.cc:290] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.522745 23641 tablet_service.cc:1505] Processing CreateTablet for tablet e194d03a6aa946fbb2c79d91b196440b (DEFAULT_TABLE table=default.Table_b6185784138b4c4b938effd9e433c447 [id=75026c75215e419983355275cf38f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.529898 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:20.529903 23646 tablet_service.cc:1505] Processing CreateTablet for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (DEFAULT_TABLE table=default.Table_a21f26a0638f42d682abb0448b2a53e5 [id=caafebb3277c4fe3b9188d8ce073ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.530309 23716 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.532703 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_21f0d4fca62a4309816edd74c6b137bf" } modify_external_catalogs: true
W20251023 20:15:20.535621 23605 leader_election.cc:343] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.536584 23637 tablet_service.cc:1505] Processing CreateTablet for tablet d62dbf044acd4459b26d2e7e50f6206e (DEFAULT_TABLE table=default.Table_8e0012945b9e401793b99da25bc5d76c [id=a0f3574408d0493e898036800ff93a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.537182 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e194d03a6aa946fbb2c79d91b196440b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.539557 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_21f0d4fca62a4309816edd74c6b137bf" } modify_external_catalogs: true
W20251023 20:15:20.539832 23135 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 473324b22da14618a84aa1136d081993 has no consensus state
I20251023 20:15:20.540958 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_441bf8050be9453c85aef2cb7ac63ea4" } modify_external_catalogs: true
I20251023 20:15:20.541417 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_441bf8050be9453c85aef2cb7ac63ea4" } modify_external_catalogs: true
I20251023 20:15:20.541667 24094 raft_consensus.cc:493] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.542076 24094 raft_consensus.cc:515] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.542399 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.543509 24094 leader_election.cc:290] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.546289 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.546718 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e418a62d00824cff89b37a7bace5c469" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.546988 23996 ts_manager.cc:194] Registered new tserver with Master: 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:20.548985 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 352812b56f8d4feb818a364d6adfd86d
I20251023 20:15:20.549389 23141 catalog_manager.cc:6719] ReplaceTablet: tablet 352812b56f8d4feb818a364d6adfd86d deleted and replaced by tablet 19da816a483a4d5b9dc768ef697d54ec
I20251023 20:15:20.549894 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 514cb3b1ed024041ac160a7d9acff618
W20251023 20:15:20.550086 23604 leader_election.cc:343] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.550205 23604 leader_election.cc:304] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.550278 23146 catalog_manager.cc:6719] ReplaceTablet: tablet 514cb3b1ed024041ac160a7d9acff618 deleted and replaced by tablet 370a790a053446b2b06314c7db242270
I20251023 20:15:20.550757 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d62dbf044acd4459b26d2e7e50f6206e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:20.551640 23604 leader_election.cc:343] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.551714 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.552250 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.552551 23145 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0f114a37952f447ab986c5799d7ff3ca" } new_table_name: "default.Table_fc71da06b58e453a9fd0589907bf8987" modify_external_catalogs: true
I20251023 20:15:20.554399 24094 raft_consensus.cc:2749] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.554569 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e1d99e08ee354f119f078584381958e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.554870 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3a387ab7d11047b392362bbe1fa7f701" } new_table_name: "default.Table_efcc5e094c4349d686c5e8af614c2c5c" modify_external_catalogs: true
W20251023 20:15:20.555064 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1d99e08ee354f119f078584381958e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.558271 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b5c5f5ccad7d42a39151aa04d42db380"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.558751 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5c5f5ccad7d42a39151aa04d42db380 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.565580 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 6480cdc1afda4671a346044f232b083e
I20251023 20:15:20.569298 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d9f73477a06542c0b215ae86716cdc4e
I20251023 20:15:20.569988 23140 master_service.cc:954] ReplaceTablet: received request to replace tablet 2fa6a838b1bd409ba2a4e2074576ee23 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.570034 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:20.570363 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:20.571341 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet fb01b3f4ff7f41e98063ebf76abeace6
W20251023 20:15:20.573071 23145 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b680b1cbc103403b91a1104248ecbdcf consensus state has no leader
I20251023 20:15:20.574625 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_2496fe6d77e44734bfced35dee7d6596"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.575203 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2496fe6d77e44734bfced35dee7d6596 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.575923 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_934c5152009843bcbf11132bbe4c8ed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.576396 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_934c5152009843bcbf11132bbe4c8ed2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.576738 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b342acb3e1f447b8985fa86b3f8c7fc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.577020 23787 tablet_bootstrap.cc:492] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.577211 23787 ts_tablet_manager.cc:1403] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.186s user 0.009s sys 0.007s
W20251023 20:15:20.577328 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.577689 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_48c1e728bef74781a6d7353e1b1181be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.578128 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48c1e728bef74781a6d7353e1b1181be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.578050 23787 raft_consensus.cc:359] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.578229 23787 raft_consensus.cc:385] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.578302 23787 raft_consensus.cc:740] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.578595 23787 consensus_queue.cc:260] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.582155 23787 ts_tablet_manager.cc:1434] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:20.582464 23787 tablet_bootstrap.cc:492] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:20.586098 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 215e21ca449c481d8716783a65105101 consensus state has no leader
I20251023 20:15:20.587728 23784 ts_tablet_manager.cc:1929] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.590091 23787 tablet_bootstrap.cc:654] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.590149 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e418a62d00824cff89b37a7bace5c469" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:20.592185 23605 leader_election.cc:343] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.592346 23605 leader_election.cc:304] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.593243 24094 raft_consensus.cc:2749] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.599247 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.600044 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.600389 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet e4bb6abe3ef848e2831839de6a28eb0f with delete_type TABLET_DATA_DELETED (replaced by tablet 8289ce37fa074891beb584bf2fff9747) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.600729 23784 log.cc:1199] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/62acc395fb294965a1ab476d37f9fab1
I20251023 20:15:20.600986 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.601459 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.601842 23784 ts_tablet_manager.cc:1950] T 62acc395fb294965a1ab476d37f9fab1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:20.602970 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.602008 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.602140 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.603291 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.602427 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.603507 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.604166 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.604346 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.604383 23364 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.604568 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.604601 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.604655 23372 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.604725 23373 tablet_service.cc:1552] Processing DeleteTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.605118 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.605664 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.605866 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 62acc395fb294965a1ab476d37f9fab1 (table default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]) successfully deleted
W20251023 20:15:20.606223 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ac55e14745534a02a2f6a422cab6b149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac55e14745534a02a2f6a422cab6b149 already in progress: creating tablet
I20251023 20:15:20.606454 23379 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.606873 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7ab2fc1342c440f9a0c55b4d52c748f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.607298 23377 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.607422 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:20.607488 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ab2fc1342c440f9a0c55b4d52c748f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.608858 23146 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_97743c4429454792b72a02f5fa2c3123" } new_table_name: "default.Table_8c497d9a1dcc41aa8466741225b4de85" modify_external_catalogs: true
I20251023 20:15:20.609326 23784 tablet_replica.cc:333] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:20.609532 23784 raft_consensus.cc:2243] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.609690 23784 raft_consensus.cc:2272] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.610200 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_de4427f05119413b8983f4e818072f98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.610711 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de4427f05119413b8983f4e818072f98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.611100 23784 ts_tablet_manager.cc:1916] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.614251 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 with delete_type TABLET_DATA_DELETED (replaced by tablet edec01a63a3046d9ae5a372fc9068b52) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.614584 23785 ts_tablet_manager.cc:1929] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:20.615900 23785 log.cc:1199] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e4bb6abe3ef848e2831839de6a28eb0f
I20251023 20:15:20.620064 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_cf2a1b576c024ec89df2ce4b0ca48a29" } new_table_name: "default.Table_a9a54be33bfd4ab79fbc7c01381625b6" modify_external_catalogs: true
I20251023 20:15:20.620986 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bcdec7c16c943079e20bb486aecb431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.620745 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.621991 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.622524 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.626845 23785 ts_tablet_manager.cc:1950] T e4bb6abe3ef848e2831839de6a28eb0f P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:20.627292 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 326) took 1417 ms. Trace:
I20251023 20:15:20.634311 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.634873 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.637456 24094 raft_consensus.cc:493] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.637811 24094 raft_consensus.cc:515] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.640188 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23225d31c2e647398aa58494c627507e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:20.641067 23604 leader_election.cc:343] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.641427 24094 leader_election.cc:290] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:20.644935 23146 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9c5b69df4c04365b1ac1457fe8300cc consensus state has no leader
I20251023 20:15:20.646060 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23225d31c2e647398aa58494c627507e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:20.647889 23605 leader_election.cc:343] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.648049 23605 leader_election.cc:304] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.648849 24094 raft_consensus.cc:2749] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:20.650252 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fea0afc8f63497d8b5cdf2352a20267 consensus state has no leader
I20251023 20:15:20.644810 23785 rpcz_store.cc:276] 1023 20:15:19.209955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.210001 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:20.013852 (+803851us) ts_tablet_manager.cc:1151] Deleting tablet e4bb6abe3ef848e2831839de6a28eb0f
1023 20:15:20.013864 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.371977 (+358113us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.611195 (+239218us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.627254 (+ 16059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":803668}
I20251023 20:15:20.651125 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet e4bb6abe3ef848e2831839de6a28eb0f (table default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]) successfully deleted
I20251023 20:15:20.651774 23143 master_service.cc:954] ReplaceTablet: received request to replace tablet 951b2276d3324c2d999c8d7fb97767bc from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.653796 23786 tablet_bootstrap.cc:492] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:20.654044 23786 ts_tablet_manager.cc:1403] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.185s user 0.008s sys 0.004s
I20251023 20:15:20.655452 23786 raft_consensus.cc:359] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.655735 23786 raft_consensus.cc:385] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.655846 23786 raft_consensus.cc:740] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.656127 23786 consensus_queue.cc:260] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.656665 23786 ts_tablet_manager.cc:1434] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:20.657033 23786 tablet_bootstrap.cc:492] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:20.664263 23786 tablet_bootstrap.cc:654] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.664543 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e338b982a564ea5b1b15c2146a96fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.667455 23787 tablet_bootstrap.cc:492] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.667622 23787 ts_tablet_manager.cc:1403] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.006s
I20251023 20:15:20.668362 23787 raft_consensus.cc:359] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.668584 23787 raft_consensus.cc:385] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.668681 23787 raft_consensus.cc:740] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.668901 23787 consensus_queue.cc:260] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.666000 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 331) took 1407 ms. Trace:
I20251023 20:15:20.669111 23785 rpcz_store.cc:276] 1023 20:15:19.244148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.244205 (+ 57us) service_pool.cc:225] Handling call
1023 20:15:20.651426 (+1407221us) ts_tablet_manager.cc:1151] Deleting tablet fe922c438a894a7aa04699b2d2897a28
1023 20:15:20.651437 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.652048 (+ 611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406959}
I20251023 20:15:20.669456 23787 ts_tablet_manager.cc:1434] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251023 20:15:20.669719 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet fe922c438a894a7aa04699b2d2897a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe922c438a894a7aa04699b2d2897a28 already in progress: creating tablet
I20251023 20:15:20.669788 23787 tablet_bootstrap.cc:492] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.670194 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 328) took 1440 ms. Trace:
I20251023 20:15:20.671409 23787 tablet_bootstrap.cc:654] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.673501 23784 ts_tablet_manager.cc:1929] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.673723 23784 log.cc:1199] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/caaccec5866a4e95afe9ccbf1f5bc494
I20251023 20:15:20.674146 23784 ts_tablet_manager.cc:1950] T caaccec5866a4e95afe9ccbf1f5bc494 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:20.675249 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet caaccec5866a4e95afe9ccbf1f5bc494 (table default.Table_fb1a22feead44f33b0a8f5f5ec849493 [id=d05778ff4e54433d9d549b7d5356afe3]) successfully deleted
I20251023 20:15:20.675611 23784 tablet_replica.cc:333] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:20.675742 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 352812b56f8d4feb818a364d6adfd86d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352812b56f8d4feb818a364d6adfd86d already in progress: creating tablet
I20251023 20:15:20.675892 23784 raft_consensus.cc:2243] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:20.675997 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
I20251023 20:15:20.676149 23784 raft_consensus.cc:2272] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.683779 23784 ts_tablet_manager.cc:1916] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.687153 23783 ts_tablet_manager.cc:1929] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.687297 23783 log.cc:1199] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/585a9af140d9479fb83d13e68a62300c
I20251023 20:15:20.687675 23783 ts_tablet_manager.cc:1950] T 585a9af140d9479fb83d13e68a62300c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:20.688416 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 340) took 1560 ms. Trace:
I20251023 20:15:20.671765 23785 rpcz_store.cc:276] 1023 20:15:19.230065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.230161 (+ 96us) service_pool.cc:225] Handling call
1023 20:15:20.670044 (+1439883us) ts_tablet_manager.cc:1151] Deleting tablet 83dae34129794f49b43038d3b1636ea0
1023 20:15:20.670054 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.670159 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1416357}
W20251023 20:15:20.691239 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 83dae34129794f49b43038d3b1636ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83dae34129794f49b43038d3b1636ea0 already in progress: creating tablet
I20251023 20:15:20.694144 23996 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d220aba7c9c0400398ffde917376ee48" instance_seqno: 1761250513291724) as {username='slave'} at 127.22.127.129:34937; Asking this server to re-register.
I20251023 20:15:20.694310 23148 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:20.688556 23783 rpcz_store.cc:276] 1023 20:15:19.128190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.128275 (+ 85us) service_pool.cc:225] Handling call
1023 20:15:20.487850 (+1359575us) ts_tablet_manager.cc:1151] Deleting tablet 585a9af140d9479fb83d13e68a62300c
1023 20:15:20.487861 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.628207 (+140346us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.687011 (+ 58804us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.688378 (+ 1367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359370}
I20251023 20:15:20.700178 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 585a9af140d9479fb83d13e68a62300c (table default.Table_c3d2d37dd2bd414fae4c968fc3f52a25 [id=97396cb497da42f6af247dd68249182e]) successfully deleted
I20251023 20:15:20.702580 23445 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:20.703024 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.704502 23996 ts_manager.cc:194] Registered new tserver with Master: d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:20.705615 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 341) took 1531 ms. Trace:
I20251023 20:15:20.706959 23783 rpcz_store.cc:276] 1023 20:15:19.174175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.174230 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:20.705441 (+1531211us) ts_tablet_manager.cc:1151] Deleting tablet 33b5efb30b5c491c8eafe3ac25688843
1023 20:15:20.705454 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.705577 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1530972}
W20251023 20:15:20.707396 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 33b5efb30b5c491c8eafe3ac25688843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33b5efb30b5c491c8eafe3ac25688843 already in progress: creating tablet
I20251023 20:15:20.709245 23784 ts_tablet_manager.cc:1929] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.709674 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 333) took 1449 ms. Trace:
I20251023 20:15:20.709807 23785 rpcz_store.cc:276] 1023 20:15:19.259776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.259833 (+ 57us) service_pool.cc:225] Handling call
1023 20:15:20.709483 (+1449650us) ts_tablet_manager.cc:1151] Deleting tablet cbacaf2fea24405caaf84d59b625fca3
1023 20:15:20.709494 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.709636 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449434}
I20251023 20:15:20.710086 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 343) took 1511 ms. Trace:
W20251023 20:15:20.710264 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet cbacaf2fea24405caaf84d59b625fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbacaf2fea24405caaf84d59b625fca3 already in progress: creating tablet
I20251023 20:15:20.710208 23783 rpcz_store.cc:276] 1023 20:15:19.198655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.198718 (+ 63us) service_pool.cc:225] Handling call
1023 20:15:20.709938 (+1511220us) ts_tablet_manager.cc:1151] Deleting tablet 83dae34129794f49b43038d3b1636ea0
1023 20:15:20.709950 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.710052 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510957}
W20251023 20:15:20.710572 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 83dae34129794f49b43038d3b1636ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83dae34129794f49b43038d3b1636ea0 already in progress: creating tablet
I20251023 20:15:20.711808 23784 log.cc:1199] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e4bb6abe3ef848e2831839de6a28eb0f
I20251023 20:15:20.712352 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 347) took 1466 ms. Trace:
I20251023 20:15:20.712471 23783 rpcz_store.cc:276] 1023 20:15:19.245734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.245787 (+ 53us) service_pool.cc:225] Handling call
1023 20:15:20.712205 (+1466418us) ts_tablet_manager.cc:1151] Deleting tablet fe922c438a894a7aa04699b2d2897a28
1023 20:15:20.712216 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.712319 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1466215}
W20251023 20:15:20.712872 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fe922c438a894a7aa04699b2d2897a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe922c438a894a7aa04699b2d2897a28 already in progress: creating tablet
I20251023 20:15:20.713485 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 336) took 1447 ms. Trace:
I20251023 20:15:20.713609 23785 rpcz_store.cc:276] 1023 20:15:19.265739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.265812 (+ 73us) service_pool.cc:225] Handling call
1023 20:15:20.713344 (+1447532us) ts_tablet_manager.cc:1151] Deleting tablet 5e73ae95c73945f081c298f49d47c904
1023 20:15:20.713353 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.713447 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1447313}
I20251023 20:15:20.713995 23784 ts_tablet_manager.cc:1950] T e4bb6abe3ef848e2831839de6a28eb0f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:20.714040 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 5e73ae95c73945f081c298f49d47c904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e73ae95c73945f081c298f49d47c904 already in progress: creating tablet
I20251023 20:15:20.714442 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 349) took 1468 ms. Trace:
I20251023 20:15:20.714560 23783 rpcz_store.cc:276] 1023 20:15:19.246035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.246073 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:20.714283 (+1468210us) ts_tablet_manager.cc:1151] Deleting tablet cbacaf2fea24405caaf84d59b625fca3
1023 20:15:20.714293 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.714411 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468033}
W20251023 20:15:20.714884 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cbacaf2fea24405caaf84d59b625fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbacaf2fea24405caaf84d59b625fca3 already in progress: creating tablet
I20251023 20:15:20.716218 23783 tablet_replica.cc:333] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.716317 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet e4bb6abe3ef848e2831839de6a28eb0f (table default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]) successfully deleted
I20251023 20:15:20.716421 23783 raft_consensus.cc:2243] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.716596 23783 raft_consensus.cc:2272] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.717150 23783 ts_tablet_manager.cc:1916] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:20.718951 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
W20251023 20:15:20.719548 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cec51c54ea534b1da4b1d170e79d9f1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cec51c54ea534b1da4b1d170e79d9f1c already in progress: creating tablet
W20251023 20:15:20.721056 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 951b2276d3324c2d999c8d7fb97767bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 951b2276d3324c2d999c8d7fb97767bc already in progress: creating tablet
W20251023 20:15:20.721700 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
I20251023 20:15:20.722168 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_2f964a2f8a294783802fdf58c31376c0" } new_table_name: "default.Table_6d6d11ad70eb49409020a15552192008" modify_external_catalogs: true
W20251023 20:15:20.725973 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
I20251023 20:15:20.726362 23784 tablet_replica.cc:333] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:20.726627 23784 raft_consensus.cc:2243] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.726837 23784 raft_consensus.cc:2272] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.727655 23784 ts_tablet_manager.cc:1916] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.729564 23785 tablet_replica.cc:333] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:20.729809 23785 raft_consensus.cc:2243] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.730007 23785 raft_consensus.cc:2272] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.730592 23785 ts_tablet_manager.cc:1916] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.759478 24159 tablet.cc:1722] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:20.767737 23147 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
W20251023 20:15:20.767705 23604 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 49053.7us
I20251023 20:15:20.768913 23790 tablet_bootstrap.cc:492] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:20.769100 23790 ts_tablet_manager.cc:1403] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.263s user 0.002s sys 0.012s
I20251023 20:15:20.770052 23790 raft_consensus.cc:359] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.770318 23790 raft_consensus.cc:385] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.770457 23790 raft_consensus.cc:740] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.770757 23790 consensus_queue.cc:260] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.774297 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0d31043ac68d4a0394d7ba5b60d539c8" } modify_external_catalogs: true
I20251023 20:15:20.774533 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_0d31043ac68d4a0394d7ba5b60d539c8" } modify_external_catalogs: true
I20251023 20:15:20.774868 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_db52858e57b64c15a85a784ef9bff9b4" } modify_external_catalogs: true
I20251023 20:15:20.775060 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_db52858e57b64c15a85a784ef9bff9b4" } modify_external_catalogs: true
I20251023 20:15:20.775215 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_9899db7c169244f38fdb46b1fd263e1a" } modify_external_catalogs: true
I20251023 20:15:20.775378 23790 ts_tablet_manager.cc:1434] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251023 20:15:20.775070 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7b6d17b4f21740f0a5dadc35a23c919f" } modify_external_catalogs: true
I20251023 20:15:20.775549 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7b6d17b4f21740f0a5dadc35a23c919f" } modify_external_catalogs: true
I20251023 20:15:20.775609 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_5d14af3d31254be5a7ff6cb5d1fea162" } modify_external_catalogs: true
I20251023 20:15:20.775755 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_5d14af3d31254be5a7ff6cb5d1fea162" } modify_external_catalogs: true
I20251023 20:15:20.775409 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_9899db7c169244f38fdb46b1fd263e1a" } modify_external_catalogs: true
I20251023 20:15:20.779536 23790 tablet_bootstrap.cc:492] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:20.780723 24157 tablet.cc:1722] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:20.781948 23784 ts_tablet_manager.cc:1929] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.782177 23784 log.cc:1199] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/2e338b982a564ea5b1b15c2146a96fa2
I20251023 20:15:20.782596 23784 ts_tablet_manager.cc:1950] T 2e338b982a564ea5b1b15c2146a96fa2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:20.786756 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 2e338b982a564ea5b1b15c2146a96fa2 (table default.Table_4f134fc02d1c4a53a726595803ef505c [id=ff3dd4d1748144c492b64ccde7297729]) successfully deleted
I20251023 20:15:20.783488 23786 tablet_bootstrap.cc:492] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:20.787168 23786 ts_tablet_manager.cc:1403] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.130s user 0.007s sys 0.006s
I20251023 20:15:20.789849 23786 raft_consensus.cc:359] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.790112 23786 raft_consensus.cc:385] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.796275 23145 master_service.cc:954] ReplaceTablet: received request to replace tablet 9fc7e18054c440098dc2470143ea7fae from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.799084 24160 raft_consensus.cc:493] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.799414 24160 raft_consensus.cc:515] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.799214 23786 raft_consensus.cc:740] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.800716 23786 consensus_queue.cc:260] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.801287 23786 ts_tablet_manager.cc:1434] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251023 20:15:20.801599 23786 tablet_bootstrap.cc:492] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:20.806005 23783 ts_tablet_manager.cc:1929] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.806175 23783 log.cc:1199] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/3894dd4feeea4c16861a6c9520059474
I20251023 20:15:20.806545 23783 ts_tablet_manager.cc:1950] T 3894dd4feeea4c16861a6c9520059474 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:20.807305 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 346) took 1569 ms. Trace:
I20251023 20:15:20.807404 23783 rpcz_store.cc:276] 1023 20:15:19.237892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.238200 (+ 308us) service_pool.cc:225] Handling call
1023 20:15:20.716160 (+1477960us) ts_tablet_manager.cc:1151] Deleting tablet 3894dd4feeea4c16861a6c9520059474
1023 20:15:20.716169 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.744195 (+ 28026us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.804098 (+ 59903us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.807251 (+ 3153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1462118}
I20251023 20:15:20.807806 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 3894dd4feeea4c16861a6c9520059474 (table default.Table_67a76ff63d4d411582ab8e10ac4f6924 [id=fb6dc9a69da44f35a0184490d90a1e5c]) successfully deleted
I20251023 20:15:20.807890 24160 leader_election.cc:290] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:20.808611 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90d58bceecba442898a3d742fa49a4a2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.808884 23786 tablet_bootstrap.cc:654] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:20.809370 23604 leader_election.cc:343] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.810446 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.810803 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90d58bceecba442898a3d742fa49a4a2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:20.811334 23605 leader_election.cc:343] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:20.811467 23605 leader_election.cc:304] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:20.812021 24160 raft_consensus.cc:2749] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:20.813850 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 352) took 1557 ms. Trace:
I20251023 20:15:20.813959 23783 rpcz_store.cc:276] 1023 20:15:19.256094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.256932 (+ 838us) service_pool.cc:225] Handling call
1023 20:15:20.813179 (+1556247us) ts_tablet_manager.cc:1151] Deleting tablet 5e73ae95c73945f081c298f49d47c904
1023 20:15:20.813190 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.813300 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491053}
W20251023 20:15:20.814447 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5e73ae95c73945f081c298f49d47c904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e73ae95c73945f081c298f49d47c904 already in progress: creating tablet
I20251023 20:15:20.815465 23787 tablet_bootstrap.cc:492] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:20.816586 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01fa5f9c700b458ea147373d78551573 has no consensus state
I20251023 20:15:20.816609 23787 ts_tablet_manager.cc:1403] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.147s user 0.007s sys 0.005s
I20251023 20:15:20.817402 23787 raft_consensus.cc:359] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.817618 23787 raft_consensus.cc:385] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.817821 23787 raft_consensus.cc:740] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.818058 23787 consensus_queue.cc:260] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.818522 23787 ts_tablet_manager.cc:1434] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:20.818866 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 374) took 1258 ms. Trace:
I20251023 20:15:20.818955 23783 rpcz_store.cc:276] 1023 20:15:19.560706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.560864 (+ 158us) service_pool.cc:225] Handling call
1023 20:15:20.818714 (+1257850us) ts_tablet_manager.cc:1151] Deleting tablet 7301d6121c9d4c8dbc642e895aa4e9f6
1023 20:15:20.818722 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.818821 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257436}
W20251023 20:15:20.819371 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7301d6121c9d4c8dbc642e895aa4e9f6 already in progress: creating tablet
I20251023 20:15:20.819790 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 372) took 1263 ms. Trace:
I20251023 20:15:20.819883 23783 rpcz_store.cc:276] 1023 20:15:19.556231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.556362 (+ 131us) service_pool.cc:225] Handling call
1023 20:15:20.819636 (+1263274us) ts_tablet_manager.cc:1151] Deleting tablet 1a64a7c2e1f242e1b54243489622bae5
1023 20:15:20.819646 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.819735 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1243925}
W20251023 20:15:20.820266 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1a64a7c2e1f242e1b54243489622bae5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a64a7c2e1f242e1b54243489622bae5 already in progress: creating tablet
I20251023 20:15:20.820624 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 373) took 1260 ms. Trace:
I20251023 20:15:20.820720 23783 rpcz_store.cc:276] 1023 20:15:19.560264 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.560319 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:20.820477 (+1260158us) ts_tablet_manager.cc:1151] Deleting tablet a21ec49194e24723b31fd9c86f1f85e4
1023 20:15:20.820486 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.820574 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244602}
W20251023 20:15:20.821058 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a21ec49194e24723b31fd9c86f1f85e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a21ec49194e24723b31fd9c86f1f85e4 already in progress: creating tablet
I20251023 20:15:20.821485 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 377) took 1214 ms. Trace:
I20251023 20:15:20.821590 23783 rpcz_store.cc:276] 1023 20:15:19.606520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.622568 (+ 16048us) service_pool.cc:225] Handling call
1023 20:15:20.821271 (+1198703us) ts_tablet_manager.cc:1151] Deleting tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0
1023 20:15:20.821280 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.821360 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155272}
W20251023 20:15:20.822010 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 already in progress: creating tablet
I20251023 20:15:20.824553 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 379) took 1145 ms. Trace:
I20251023 20:15:20.824652 23783 rpcz_store.cc:276] 1023 20:15:19.679415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.680376 (+ 961us) service_pool.cc:225] Handling call
1023 20:15:20.824387 (+1144011us) ts_tablet_manager.cc:1151] Deleting tablet b90810781f724c78935a33d868ffa27d
1023 20:15:20.824397 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.824488 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143612}
W20251023 20:15:20.825044 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
I20251023 20:15:20.827811 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 381) took 1129 ms. Trace:
I20251023 20:15:20.827909 23783 rpcz_store.cc:276] 1023 20:15:19.697960 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.700787 (+ 2827us) service_pool.cc:225] Handling call
1023 20:15:20.827652 (+1126865us) ts_tablet_manager.cc:1151] Deleting tablet 6a54470d088748b0be22ab9db5a25f87
1023 20:15:20.827663 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.827758 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126646}
I20251023 20:15:20.829056 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 380) took 1131 ms. Trace:
I20251023 20:15:20.829209 23783 rpcz_store.cc:276] 1023 20:15:19.697819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.709541 (+ 11722us) service_pool.cc:225] Handling call
1023 20:15:20.828904 (+1119363us) ts_tablet_manager.cc:1151] Deleting tablet 25079532547d4175802b84877790662d
1023 20:15:20.828914 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.828998 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119142}
W20251023 20:15:20.830332 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 25079532547d4175802b84877790662d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25079532547d4175802b84877790662d already in progress: creating tablet
I20251023 20:15:20.830793 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 382) took 1096 ms. Trace:
I20251023 20:15:20.830883 23783 rpcz_store.cc:276] 1023 20:15:19.733878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.733940 (+ 62us) service_pool.cc:225] Handling call
1023 20:15:20.830647 (+1096707us) ts_tablet_manager.cc:1151] Deleting tablet 925e53d9ea6f4c7baa38551eb917828d
1023 20:15:20.830656 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.830746 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096449}
W20251023 20:15:20.831254 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 925e53d9ea6f4c7baa38551eb917828d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 925e53d9ea6f4c7baa38551eb917828d already in progress: creating tablet
I20251023 20:15:20.832095 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 383) took 1088 ms. Trace:
I20251023 20:15:20.832190 23783 rpcz_store.cc:276] 1023 20:15:19.743328 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.743389 (+ 61us) service_pool.cc:225] Handling call
1023 20:15:20.831961 (+1088572us) ts_tablet_manager.cc:1151] Deleting tablet 5346a221c78342359eb6e418bd12cd24
1023 20:15:20.831970 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.832050 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1088304}
W20251023 20:15:20.832626 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5346a221c78342359eb6e418bd12cd24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5346a221c78342359eb6e418bd12cd24 already in progress: creating tablet
I20251023 20:15:20.833655 23787 tablet_bootstrap.cc:492] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.835458 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 384) took 1023 ms. Trace:
I20251023 20:15:20.835565 23783 rpcz_store.cc:276] 1023 20:15:19.812178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.812236 (+ 58us) service_pool.cc:225] Handling call
1023 20:15:20.835288 (+1023052us) ts_tablet_manager.cc:1151] Deleting tablet 7540354f8577482785e6341fa9022315
1023 20:15:20.835296 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.835410 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022343}
W20251023 20:15:20.835948 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7540354f8577482785e6341fa9022315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7540354f8577482785e6341fa9022315 already in progress: creating tablet
I20251023 20:15:20.838591 23786 tablet_bootstrap.cc:492] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:20.838774 23786 ts_tablet_manager.cc:1403] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.002s
I20251023 20:15:20.841655 23790 tablet_bootstrap.cc:654] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.842319 23148 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:20.843106 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d39a64892c5b49d99dccfbe4163d6f53" } new_table_name: "default.Table_62817ee2bac14a6a8c254e60d7836061" modify_external_catalogs: true
I20251023 20:15:20.844966 23145 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_16e3b6fb0a35485b81cea711419e6295" } new_table_name: "default.Table_0ac0b8ae289f41e391d7d9b51995bd28" modify_external_catalogs: true
I20251023 20:15:20.853387 23787 tablet_bootstrap.cc:654] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:20.855545 23145 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0150ab3778c34c77a7e772dd6e82b32f consensus state has no leader
I20251023 20:15:20.858624 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_c67489b303534a86b98a072da3af3ec1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.859200 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c67489b303534a86b98a072da3af3ec1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.865113 23144 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e418a62d00824cff89b37a7bace5c469 (table default.Table_3a893d487d5b46ca84e6d00ce6cf7137 [id=751ae1aba5564ffa8dd6274778d1b8c7]). Expected version 1 got 0
I20251023 20:15:20.865410 23144 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:20.865624 23144 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_01e28a3f1691446384d30911e4894273 [id=b0c6ab2358b84be2862d31f863e3c32d]). Expected version 3 got 0
I20251023 20:15:20.865849 23144 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]). Expected version 3 got 0
W20251023 20:15:20.867197 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a630b83f614447ebf2a08851d6b827d consensus state has no leader
I20251023 20:15:20.867594 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7a801afdfd4b4207b0e87c57ee93e01b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.868224 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a801afdfd4b4207b0e87c57ee93e01b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.868855 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_3a96b5d4c58047638a88c97bb9529c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.869341 23785 ts_tablet_manager.cc:1929] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.869536 23785 log.cc:1199] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/c95db7b5b9fd46acae8ffa7119915907
I20251023 20:15:20.870574 23786 raft_consensus.cc:359] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.870787 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a86a25659ec44d7c825783036342ed45
I20251023 20:15:20.870800 23786 raft_consensus.cc:385] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.870883 23786 raft_consensus.cc:740] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.870939 23785 ts_tablet_manager.cc:1950] T c95db7b5b9fd46acae8ffa7119915907 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:20.871131 23786 consensus_queue.cc:260] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.871245 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.871383 23141 master_service.cc:954] ReplaceTablet: received request to replace tablet 40e0301fb5974ec5b6e8291230a736c2 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.871475 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.871658 23786 ts_tablet_manager.cc:1434] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20251023 20:15:20.871659 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.871824 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.872020 23786 tablet_bootstrap.cc:492] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:20.872498 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:20.872735 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.873103 23146 master_service.cc:954] ReplaceTablet: received request to replace tablet 9d1ddde7e3134e23b3a9356523a93753 from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:20.873674 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_31442b53b2f748d0895b0014aa8ae96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.873874 23786 tablet_bootstrap.cc:654] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:20.874238 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31442b53b2f748d0895b0014aa8ae96f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:20.874488 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a86a25659ec44d7c825783036342ed45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a86a25659ec44d7c825783036342ed45 already in progress: creating tablet
I20251023 20:15:20.875084 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.875175 24157 tablet.cc:1717] T 040a44e272b7410c98c109fbceb19192 P 9bc6522db11e4960a9aad84435371a32: Skipping requested alter to schema version 4, tablet already version 4
I20251023 20:15:20.875725 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 2fa6a838b1bd409ba2a4e2074576ee23
I20251023 20:15:20.876168 23140 catalog_manager.cc:6719] ReplaceTablet: tablet 2fa6a838b1bd409ba2a4e2074576ee23 deleted and replaced by tablet aa59141039de402e9ccce427604ecb31
I20251023 20:15:20.876286 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.876490 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.877732 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a3c5a2b670b04aeca43dac4bab866e54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.878039 23787 tablet_bootstrap.cc:492] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:20.878182 23787 ts_tablet_manager.cc:1403] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.002s
W20251023 20:15:20.878228 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3c5a2b670b04aeca43dac4bab866e54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.879082 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 385) took 1006 ms. Trace:
I20251023 20:15:20.879155 23787 raft_consensus.cc:359] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.879338 23787 raft_consensus.cc:385] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.879252 23783 rpcz_store.cc:276] 1023 20:15:19.872611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.872666 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:20.878900 (+1006234us) ts_tablet_manager.cc:1151] Deleting tablet c9bfbf0acdc84f2f8fe553694ee9897c
1023 20:15:20.878910 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.879027 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005990}
I20251023 20:15:20.879427 23787 raft_consensus.cc:740] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.880012 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 344) took 1506 ms. Trace:
I20251023 20:15:20.880116 23785 rpcz_store.cc:276] 1023 20:15:19.373987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.374045 (+ 58us) service_pool.cc:225] Handling call
1023 20:15:20.729474 (+1355429us) ts_tablet_manager.cc:1151] Deleting tablet c95db7b5b9fd46acae8ffa7119915907
1023 20:15:20.729487 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.793280 (+ 63793us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.869186 (+ 75906us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:20.879955 (+ 10769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355080}
I20251023 20:15:20.880319 23783 tablet_replica.cc:333] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.880507 23783 raft_consensus.cc:2243] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:20.880481 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 79e3d422a66f432789c0cf4a00e57af2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e3d422a66f432789c0cf4a00e57af2 already in progress: creating tablet
I20251023 20:15:20.880683 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 356) took 1334 ms. Trace:
I20251023 20:15:20.880748 23783 pending_rounds.cc:70] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Trying to abort 1 pending ops.
I20251023 20:15:20.880816 23783 pending_rounds.cc:77] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082130458718208 op_type: NO_OP noop_request { }
I20251023 20:15:20.880927 23783 raft_consensus.cc:2889] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:20.881016 23783 raft_consensus.cc:2272] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.881323 23509 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f766bfe7dd94352abcaf34a200d00bc with delete_type TABLET_DATA_DELETED (replaced by tablet 53f33020319d48c791e1156c67b55be6) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:20.881506 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a96b5d4c58047638a88c97bb9529c22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.881893 23783 ts_tablet_manager.cc:1916] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.882117 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.882488 23784 tablet_replica.cc:333] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:20.882666 23784 raft_consensus.cc:2243] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.882822 23784 raft_consensus.cc:2272] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.880774 23785 rpcz_store.cc:276] 1023 20:15:19.546238 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.546313 (+ 75us) service_pool.cc:225] Handling call
1023 20:15:20.880525 (+1334212us) ts_tablet_manager.cc:1151] Deleting tablet 1a64a7c2e1f242e1b54243489622bae5
1023 20:15:20.880535 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.880630 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320990}
I20251023 20:15:20.886030 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 365) took 1132 ms. Trace:
I20251023 20:15:20.888641 23785 rpcz_store.cc:276] 1023 20:15:19.753190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.753605 (+ 415us) service_pool.cc:225] Handling call
1023 20:15:20.885841 (+1132236us) ts_tablet_manager.cc:1151] Deleting tablet 5346a221c78342359eb6e418bd12cd24
1023 20:15:20.885852 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.885978 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131961}
I20251023 20:15:20.889102 23783 ts_tablet_manager.cc:1929] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:20.889256 23783 log.cc:1199] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/7f766bfe7dd94352abcaf34a200d00bc
I20251023 20:15:20.889683 23783 ts_tablet_manager.cc:1950] T 7f766bfe7dd94352abcaf34a200d00bc P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:20.890570 23790 tablet_bootstrap.cc:492] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:20.890961 23790 ts_tablet_manager.cc:1403] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.112s user 0.001s sys 0.004s
I20251023 20:15:20.891012 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.891855 23783 tablet_replica.cc:333] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.891829 23790 raft_consensus.cc:359] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.892023 23790 raft_consensus.cc:385] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.892050 23783 raft_consensus.cc:2243] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.892081 23790 raft_consensus.cc:740] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.892230 23783 raft_consensus.cc:2272] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.892289 23790 consensus_queue.cc:260] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.892746 23783 ts_tablet_manager.cc:1916] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:20.894716 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
W20251023 20:15:20.895037 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 48436f3d8f1443db990a2d10b668b70a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48436f3d8f1443db990a2d10b668b70a already in progress: creating tablet
W20251023 20:15:20.895318 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 46f700ea3db04a4bad539c943e850a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f700ea3db04a4bad539c943e850a44 already in progress: creating tablet
I20251023 20:15:20.895237 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 92576756ec574ceaa4c81f435cb72301 (DEFAULT_TABLE table=default.Table_eb73fed1f9bb425daff2e3278ad80ece [id=fba1e090c19d406c8c9c4728885f1e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.895666 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92576756ec574ceaa4c81f435cb72301. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:20.895668 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:20.895884 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7645c36c983f438f8b1223256fd3ad2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7645c36c983f438f8b1223256fd3ad2c already in progress: creating tablet
I20251023 20:15:20.896039 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 7f766bfe7dd94352abcaf34a200d00bc (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]) successfully deleted
W20251023 20:15:20.896276 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ac55e14745534a02a2f6a422cab6b149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac55e14745534a02a2f6a422cab6b149 already in progress: creating tablet
I20251023 20:15:20.896395 23790 ts_tablet_manager.cc:1434] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
W20251023 20:15:20.896468 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 352812b56f8d4feb818a364d6adfd86d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352812b56f8d4feb818a364d6adfd86d already in progress: creating tablet
I20251023 20:15:20.898236 23645 tablet_service.cc:1505] Processing CreateTablet for tablet dae906f257c346d7a1b28c9052d600b5 (DEFAULT_TABLE table=default.Table_775428d3af964d538226c836574a899d [id=818f0fca06904f09ad46c9e184c982b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.898627 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dae906f257c346d7a1b28c9052d600b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.899099 23790 tablet_bootstrap.cc:492] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:20.900768 23790 tablet_bootstrap.cc:654] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.901093 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 8e718cb7205643fd8c3045c3fab30d3e (DEFAULT_TABLE table=default.Table_464dc629d50b4995b30ef5c463e9f316 [id=8477c10fb2f4422996d4eab8717d7fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.901472 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e718cb7205643fd8c3045c3fab30d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.901768 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.902134 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 363) took 1149 ms. Trace:
I20251023 20:15:20.902220 23785 rpcz_store.cc:276] 1023 20:15:19.752908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.753775 (+ 867us) service_pool.cc:225] Handling call
1023 20:15:20.901969 (+1148194us) ts_tablet_manager.cc:1151] Deleting tablet 6a54470d088748b0be22ab9db5a25f87
1023 20:15:20.901979 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.902085 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147986}
I20251023 20:15:20.902720 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.902877 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 364) took 1149 ms. Trace:
I20251023 20:15:20.902951 23785 rpcz_store.cc:276] 1023 20:15:19.752998 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.753635 (+ 637us) service_pool.cc:225] Handling call
1023 20:15:20.902453 (+1148818us) ts_tablet_manager.cc:1151] Deleting tablet 925e53d9ea6f4c7baa38551eb917828d
1023 20:15:20.902461 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.902557 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148358}
I20251023 20:15:20.903252 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.903491 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 362) took 1150 ms. Trace:
I20251023 20:15:20.903553 23507 tablet_service.cc:1505] Processing CreateTablet for tablet a744214a24cc4bf28654ffa417ce0d3d (DEFAULT_TABLE table=default.Table_273b73dbda4e41aea636fdc95149120a [id=124ec9a7c9574e54b885ca5b31aec53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.903578 23785 rpcz_store.cc:276] 1023 20:15:19.752823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.756861 (+ 4038us) service_pool.cc:225] Handling call
1023 20:15:20.903165 (+1146304us) ts_tablet_manager.cc:1151] Deleting tablet 25079532547d4175802b84877790662d
1023 20:15:20.903368 (+ 203us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.903445 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142838}
I20251023 20:15:20.903954 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 360) took 1151 ms. Trace:
I20251023 20:15:20.904695 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 01fa5f9c700b458ea147373d78551573 (DEFAULT_TABLE table=default.Table_6d6d11ad70eb49409020a15552192008 [id=9ecd31de55764cc9ab7f83b1a614302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.904865 23378 tablet_service.cc:1505] Processing CreateTablet for tablet dae906f257c346d7a1b28c9052d600b5 (DEFAULT_TABLE table=default.Table_775428d3af964d538226c836574a899d [id=818f0fca06904f09ad46c9e184c982b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.905150 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 40c659c273c449939d0dd708d39b393f (DEFAULT_TABLE table=default.Table_f21169f4f8814e739bc80b279a060a39 [id=6b4bb32ab663438da2c6b86b88796979]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.905257 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dae906f257c346d7a1b28c9052d600b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.905609 23787 consensus_queue.cc:260] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.906186 23509 tablet_service.cc:1505] Processing CreateTablet for tablet 40c659c273c449939d0dd708d39b393f (DEFAULT_TABLE table=default.Table_f21169f4f8814e739bc80b279a060a39 [id=6b4bb32ab663438da2c6b86b88796979]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.906679 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet c95db7b5b9fd46acae8ffa7119915907 (table default.Table_91596859556249fbb6e82b7dc5756a92 [id=5dff7019113a4323ae78b11f6b547b53]) successfully deleted
W20251023 20:15:20.906937 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1a64a7c2e1f242e1b54243489622bae5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a64a7c2e1f242e1b54243489622bae5 already in progress: creating tablet
I20251023 20:15:20.906977 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a744214a24cc4bf28654ffa417ce0d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.907652 23787 ts_tablet_manager.cc:1434] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20251023 20:15:20.908676 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01fa5f9c700b458ea147373d78551573. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.908942 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet bb475e303d07491bbc19dc3f628bceb7
I20251023 20:15:20.908702 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet c97e2e0f839a4ef389fe100b6a81301e
I20251023 20:15:20.908814 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet ca11ddd5d97544cf916b87fd3dff219d
I20251023 20:15:20.912695 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40c659c273c449939d0dd708d39b393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.913239 23786 tablet_bootstrap.cc:492] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:20.913393 23786 ts_tablet_manager.cc:1403] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.001s
I20251023 20:15:20.914637 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40c659c273c449939d0dd708d39b393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.918174 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 92576756ec574ceaa4c81f435cb72301 (DEFAULT_TABLE table=default.Table_eb73fed1f9bb425daff2e3278ad80ece [id=fba1e090c19d406c8c9c4728885f1e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.918591 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92576756ec574ceaa4c81f435cb72301. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.921692 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 8e718cb7205643fd8c3045c3fab30d3e (DEFAULT_TABLE table=default.Table_464dc629d50b4995b30ef5c463e9f316 [id=8477c10fb2f4422996d4eab8717d7fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.922109 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e718cb7205643fd8c3045c3fab30d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.923509 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 92576756ec574ceaa4c81f435cb72301 (DEFAULT_TABLE table=default.Table_eb73fed1f9bb425daff2e3278ad80ece [id=fba1e090c19d406c8c9c4728885f1e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.923897 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92576756ec574ceaa4c81f435cb72301. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.925246 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 8e718cb7205643fd8c3045c3fab30d3e (DEFAULT_TABLE table=default.Table_464dc629d50b4995b30ef5c463e9f316 [id=8477c10fb2f4422996d4eab8717d7fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.925653 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e718cb7205643fd8c3045c3fab30d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.925681 23637 tablet_service.cc:1505] Processing CreateTablet for tablet a744214a24cc4bf28654ffa417ce0d3d (DEFAULT_TABLE table=default.Table_273b73dbda4e41aea636fdc95149120a [id=124ec9a7c9574e54b885ca5b31aec53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.926033 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a744214a24cc4bf28654ffa417ce0d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.926524 23783 ts_tablet_manager.cc:1929] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.926648 23783 log.cc:1199] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:20.927055 23783 ts_tablet_manager.cc:1950] T 83da09ee0d10435ba89a19dc378d34eb P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:20.927950 23783 tablet_replica.cc:333] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:20.928154 23783 raft_consensus.cc:2243] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:20.928315 23783 raft_consensus.cc:2272] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:20.928841 23783 ts_tablet_manager.cc:1916] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:20.929282 23375 tablet_service.cc:1505] Processing CreateTablet for tablet a744214a24cc4bf28654ffa417ce0d3d (DEFAULT_TABLE table=default.Table_273b73dbda4e41aea636fdc95149120a [id=124ec9a7c9574e54b885ca5b31aec53f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.930143 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a744214a24cc4bf28654ffa417ce0d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.932001 23516 tablet_service.cc:1505] Processing CreateTablet for tablet dae906f257c346d7a1b28c9052d600b5 (DEFAULT_TABLE table=default.Table_775428d3af964d538226c836574a899d [id=818f0fca06904f09ad46c9e184c982b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.933413 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dae906f257c346d7a1b28c9052d600b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:20.934847 23604 consensus_peers.cc:597] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:20.936410 23786 raft_consensus.cc:359] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:20.939527 23786 raft_consensus.cc:385] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:20.942006 24098 raft_consensus.cc:493] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:20.942179 24098 raft_consensus.cc:515] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:20.943504 23786 raft_consensus.cc:740] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:20.943843 23786 consensus_queue.cc:260] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:20.945376 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 5346a221c78342359eb6e418bd12cd24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5346a221c78342359eb6e418bd12cd24 already in progress: creating tablet
W20251023 20:15:20.945784 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 925e53d9ea6f4c7baa38551eb917828d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 925e53d9ea6f4c7baa38551eb917828d already in progress: creating tablet
I20251023 20:15:20.946305 24098 leader_election.cc:290] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:20.946887 24159 tablet.cc:1717] T 040a44e272b7410c98c109fbceb19192 P d220aba7c9c0400398ffde917376ee48: Skipping requested alter to schema version 4, tablet already version 4
I20251023 20:15:20.946947 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 01fa5f9c700b458ea147373d78551573 (DEFAULT_TABLE table=default.Table_6d6d11ad70eb49409020a15552192008 [id=9ecd31de55764cc9ab7f83b1a614302f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:20.947755 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 25079532547d4175802b84877790662d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25079532547d4175802b84877790662d already in progress: creating tablet
I20251023 20:15:20.948218 23145 catalog_manager.cc:6194] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a] alter complete (version 4)
I20251023 20:15:20.948946 23786 ts_tablet_manager.cc:1434] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20251023 20:15:20.950739 23787 tablet_bootstrap.cc:492] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:20.951213 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01fa5f9c700b458ea147373d78551573. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.952059 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e31aebfc87734ffeb2c0b25c07cb3f70" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:20.952330 23784 ts_tablet_manager.cc:1916] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:20.953747 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
W20251023 20:15:20.953979 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 951b2276d3324c2d999c8d7fb97767bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 951b2276d3324c2d999c8d7fb97767bc already in progress: creating tablet
W20251023 20:15:20.954177 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cec51c54ea534b1da4b1d170e79d9f1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cec51c54ea534b1da4b1d170e79d9f1c already in progress: creating tablet
W20251023 20:15:20.954327 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
W20251023 20:15:20.954519 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
W20251023 20:15:20.954701 23133 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
W20251023 20:15:20.954907 23133 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1bcdec7c16c943079e20bb486aecb431 because the tablet was not found. No further retry: Not found: Tablet not found: 1bcdec7c16c943079e20bb486aecb431
I20251023 20:15:20.955127 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 83da09ee0d10435ba89a19dc378d34eb (table default.Table_a21f26a0638f42d682abb0448b2a53e5 [id=caafebb3277c4fe3b9188d8ce073ca55]) successfully deleted
I20251023 20:15:20.956832 23786 tablet_bootstrap.cc:492] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:20.962785 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a (DEFAULT_TABLE table=default.Table_d82a467c784f46dab749a2bfaa0aedbe [id=f6d53450bf2e46678751ec0bc88efcde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.966172 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 26a680ad3aed487593bd1b63004dbbfc (DEFAULT_TABLE table=default.Table_86c8c43350214e2eb31dc3b2e1f5333b [id=d449a30a75d7469d877e700b37b034dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.966528 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 53f33020319d48c791e1156c67b55be6 (DEFAULT_TABLE table=default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.971071 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a62c97e0a3459e91c24c1c2f92389a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.971601 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26a680ad3aed487593bd1b63004dbbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.972105 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.972280 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 1b220ceff0924b26ae8a54dfb8705456 (DEFAULT_TABLE table=default.Table_b9e2bcb23af94936a9fbfd0817d9a51e [id=3528c0390b7240e8a565945035555dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.973395 23786 tablet_bootstrap.cc:654] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:20.973402 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 53f33020319d48c791e1156c67b55be6 (DEFAULT_TABLE table=default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.973855 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53f33020319d48c791e1156c67b55be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.976130 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53f33020319d48c791e1156c67b55be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.904042 23785 rpcz_store.cc:276] 1023 20:15:19.752608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.761149 (+ 8541us) service_pool.cc:225] Handling call
1023 20:15:20.903800 (+1142651us) ts_tablet_manager.cc:1151] Deleting tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0
1023 20:15:20.903808 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.903902 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142396}
I20251023 20:15:20.978017 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a (DEFAULT_TABLE table=default.Table_d82a467c784f46dab749a2bfaa0aedbe [id=f6d53450bf2e46678751ec0bc88efcde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.978510 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 26a680ad3aed487593bd1b63004dbbfc (DEFAULT_TABLE table=default.Table_86c8c43350214e2eb31dc3b2e1f5333b [id=d449a30a75d7469d877e700b37b034dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.957682 23783 ts_tablet_manager.cc:1929] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:20.964061 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 40c659c273c449939d0dd708d39b393f (DEFAULT_TABLE table=default.Table_f21169f4f8814e739bc80b279a060a39 [id=6b4bb32ab663438da2c6b86b88796979]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.964623 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 01fa5f9c700b458ea147373d78551573 (DEFAULT_TABLE table=default.Table_6d6d11ad70eb49409020a15552192008 [id=9ecd31de55764cc9ab7f83b1a614302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.966370 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 53f33020319d48c791e1156c67b55be6 (DEFAULT_TABLE table=default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.979272 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.979285 23505 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.967178 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a (DEFAULT_TABLE table=default.Table_d82a467c784f46dab749a2bfaa0aedbe [id=f6d53450bf2e46678751ec0bc88efcde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.968693 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 26a680ad3aed487593bd1b63004dbbfc (DEFAULT_TABLE table=default.Table_86c8c43350214e2eb31dc3b2e1f5333b [id=d449a30a75d7469d877e700b37b034dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.969144 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.979453 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:20.969197 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 1b220ceff0924b26ae8a54dfb8705456 (DEFAULT_TABLE table=default.Table_b9e2bcb23af94936a9fbfd0817d9a51e [id=3528c0390b7240e8a565945035555dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.969458 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.970193 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.970712 23640 tablet_service.cc:1505] Processing CreateTablet for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2 (DEFAULT_TABLE table=default.Table_322349f14bf94a7cbd9cb6cb92793905 [id=9ae6a893fdef4f328cbfd0ee2d57f5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.970921 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 473324b22da14618a84aa1136d081993 (DEFAULT_TABLE table=default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.972462 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 82ae1a94b011412c851cd96946744fc4 (DEFAULT_TABLE table=default.Table_654a302882be43cd867d1b1ad3dc8bce [id=c830d9df19924cb09bb38ff2d445b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.979671 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b220ceff0924b26ae8a54dfb8705456. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:20.972688 23942 consensus_queue.cc:1048] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:20.984550 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a0beeea080b946158748ce5a6db61212"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.985205 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0beeea080b946158748ce5a6db61212 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.986232 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 1b220ceff0924b26ae8a54dfb8705456 (DEFAULT_TABLE table=default.Table_b9e2bcb23af94936a9fbfd0817d9a51e [id=3528c0390b7240e8a565945035555dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.986503 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:20.986505 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6311c71cac724f7f8ba71a660247f056"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:20.987010 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6311c71cac724f7f8ba71a660247f056 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.988588 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d0548a52907a4b1984665988fa00b425"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:20.988989 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:20.989099 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0548a52907a4b1984665988fa00b425 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:20.989482 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 82ae1a94b011412c851cd96946744fc4 (DEFAULT_TABLE table=default.Table_654a302882be43cd867d1b1ad3dc8bce [id=c830d9df19924cb09bb38ff2d445b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.990075 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2 (DEFAULT_TABLE table=default.Table_322349f14bf94a7cbd9cb6cb92793905 [id=9ae6a893fdef4f328cbfd0ee2d57f5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.990236 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8370048ef7d8400c9ea8f72c19305e61" } new_table_name: "default.Table_7cf41fd1dc6f47e5982f11504a3adf74" modify_external_catalogs: true
W20251023 20:15:20.991379 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 already in progress: creating tablet
I20251023 20:15:20.993167 23139 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]). Expected version 2 got 0
I20251023 20:15:20.993417 23139 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]). Expected version 1 got 0
I20251023 20:15:20.993405 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2 (DEFAULT_TABLE table=default.Table_322349f14bf94a7cbd9cb6cb92793905 [id=9ae6a893fdef4f328cbfd0ee2d57f5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:20.997735 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 473324b22da14618a84aa1136d081993 (DEFAULT_TABLE table=default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.000584 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40c659c273c449939d0dd708d39b393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.000602 23783 log.cc:1199] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/23225d31c2e647398aa58494c627507e
I20251023 20:15:21.001011 23942 raft_consensus.cc:493] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.001195 23942 raft_consensus.cc:515] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.001935 23942 leader_election.cc:290] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.003165 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:20.996961 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e6e2be420a6342bc9d1311bd4fe6080c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.003684 23783 ts_tablet_manager.cc:1950] T 23225d31c2e647398aa58494c627507e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:21.004004 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6e2be420a6342bc9d1311bd4fe6080c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.004855 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5368813164949ff80e27bf59dcc931a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:21.005843 23604 leader_election.cc:343] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.006660 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 23225d31c2e647398aa58494c627507e (table default.Table_67a7910474894d659750844c77625a9d [id=82c65a189d13499c8471cb1ce0894c45]) successfully deleted
I20251023 20:15:21.007040 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 82ae1a94b011412c851cd96946744fc4 (DEFAULT_TABLE table=default.Table_654a302882be43cd867d1b1ad3dc8bce [id=c830d9df19924cb09bb38ff2d445b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.007233 23783 tablet_replica.cc:333] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:21.007673 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 473324b22da14618a84aa1136d081993 (DEFAULT_TABLE table=default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.007782 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01fa5f9c700b458ea147373d78551573. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.008631 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.009605 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ae1a94b011412c851cd96946744fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.011572 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b6185784138b4c4b938effd9e433c447" } new_table_name: "default.Table_410a6baf06a642b18af37bebddac662f" modify_external_catalogs: true
I20251023 20:15:21.012703 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26a680ad3aed487593bd1b63004dbbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.013609 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473324b22da14618a84aa1136d081993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.014086 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 40e0301fb5974ec5b6e8291230a736c2
I20251023 20:15:21.014544 23141 catalog_manager.cc:6719] ReplaceTablet: tablet 40e0301fb5974ec5b6e8291230a736c2 deleted and replaced by tablet 4bce63866d524067b87145b07347ffcf
I20251023 20:15:21.015960 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53f33020319d48c791e1156c67b55be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.019210 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473324b22da14618a84aa1136d081993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.019129 23783 raft_consensus.cc:2243] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:21.019908 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.020104 23787 tablet_bootstrap.cc:654] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.020851 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ae1a94b011412c851cd96946744fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.023650 23509 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.025055 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.025328 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b220ceff0924b26ae8a54dfb8705456. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.025832 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.026774 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7c5125a919774476a6ec9f806c2d329e" } new_table_name: "default.Table_0e4d6c666b2d4372afd985345a92edd4" modify_external_catalogs: true
I20251023 20:15:21.033455 23794 raft_consensus.cc:493] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.034194 23794 raft_consensus.cc:515] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.035503 23794 leader_election.cc:290] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:21.039927 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d951134f6773410386bc57f7f90310c3 consensus state has no leader
W20251023 20:15:21.055527 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e194d03a6aa946fbb2c79d91b196440b consensus state has no leader
W20251023 20:15:21.068658 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 975fc4848c8f4db4bfd2dad9ac620d83 consensus state has no leader
I20251023 20:15:21.071679 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d7117a4238804d35b9c887c9d50900f4" } new_table_name: "default.Table_43421f044a504b98a442f09d4fc5c601" modify_external_catalogs: true
I20251023 20:15:21.074158 24098 raft_consensus.cc:493] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:21.074849 23333 leader_election.cc:343] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.077495 23516 tablet_service.cc:1505] Processing CreateTablet for tablet a4d71f0cfdb6449fbd74b854c15326d6 (DEFAULT_TABLE table=default.Table_c67489b303534a86b98a072da3af3ec1 [id=d6f62018af4a4ed58edf91d90e430e01]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.081163 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2ae19cc80194363b8a9c96fbafb4ebe consensus state has no leader
I20251023 20:15:21.086884 23506 tablet_service.cc:1505] Processing CreateTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 (DEFAULT_TABLE table=default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9 [id=d05789186efe404f94adbf2a543ba4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.090204 23514 tablet_service.cc:1505] Processing CreateTablet for tablet c3d145a33b964a4ca1f0001ac6cf990b (DEFAULT_TABLE table=default.Table_7ab2fc1342c440f9a0c55b4d52c748f8 [id=f59bbf36d431443a85154ff155cb0e66]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.091408 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 84096.9us
I20251023 20:15:21.091465 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.091169 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5368813164949ff80e27bf59dcc931a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.091831 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26a680ad3aed487593bd1b63004dbbfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.092177 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a62c97e0a3459e91c24c1c2f92389a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.092298 24098 raft_consensus.cc:515] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.092955 24098 leader_election.cc:290] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:21.093130 23333 leader_election.cc:343] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e31aebfc87734ffeb2c0b25c07cb3f70
I20251023 20:15:21.093256 23333 leader_election.cc:304] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.095240 24098 raft_consensus.cc:2749] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.095216 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:21.095415 24098 raft_consensus.cc:493] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.095568 24098 raft_consensus.cc:515] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.096125 24098 leader_election.cc:290] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.096796 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.096673 23783 raft_consensus.cc:2272] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:21.101058 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a62c97e0a3459e91c24c1c2f92389a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.091848 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 361) took 1236 ms. Trace:
I20251023 20:15:21.104624 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_86c8c43350214e2eb31dc3b2e1f5333b" } new_table_name: "default.Table_cca4260f97404dc2abc827888a312ee4" modify_external_catalogs: true
I20251023 20:15:21.106129 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a3c5a2b670b04aeca43dac4bab866e54" } modify_external_catalogs: true
I20251023 20:15:21.106897 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a3c5a2b670b04aeca43dac4bab866e54" } modify_external_catalogs: true
I20251023 20:15:21.106616 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c9ccf1142a9b449f91e99360376a7b22" } modify_external_catalogs: true
I20251023 20:15:21.107415 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c9ccf1142a9b449f91e99360376a7b22" } modify_external_catalogs: true
I20251023 20:15:21.108008 23650 tablet_service.cc:1505] Processing CreateTablet for tablet a4d71f0cfdb6449fbd74b854c15326d6 (DEFAULT_TABLE table=default.Table_c67489b303534a86b98a072da3af3ec1 [id=d6f62018af4a4ed58edf91d90e430e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.108824 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d71f0cfdb6449fbd74b854c15326d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.109833 23513 tablet_service.cc:1505] Processing CreateTablet for tablet fa0a46ffa626447eab4cdb956e798005 (DEFAULT_TABLE table=default.Table_934c5152009843bcbf11132bbe4c8ed2 [id=09f8335a5b03409c90559c8313f5d135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.111455 23503 tablet_service.cc:1505] Processing CreateTablet for tablet 370a790a053446b2b06314c7db242270 (DEFAULT_TABLE table=default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.112856 23509 tablet_service.cc:1505] Processing CreateTablet for tablet e9e2fec8f4e14036af714427481f2f36 (DEFAULT_TABLE table=default.Table_de4427f05119413b8983f4e818072f98 [id=591b99acc8dd4e7388acb07fdf88a50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.114970 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 2202632a57514c849760509360113078 (DEFAULT_TABLE table=default.Table_b5c5f5ccad7d42a39151aa04d42db380 [id=de3be1729400426bb9957ccede88ceac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.115494 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 5161336598884742ad403fc88c9d9f0b (DEFAULT_TABLE table=default.Table_e1d99e08ee354f119f078584381958e6 [id=a34d678a3e1f41d8b65903bc2126e4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.118366 23146 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet aafcfbd4b2f54623ab0f13215dda1e06
I20251023 20:15:21.118727 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 0858358aabff47a69dc77f4669b225c5
I20251023 20:15:21.118783 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 473324b22da14618a84aa1136d081993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.130852 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d71f0cfdb6449fbd74b854c15326d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.132102 23643 tablet_service.cc:1505] Processing CreateTablet for tablet e9e2fec8f4e14036af714427481f2f36 (DEFAULT_TABLE table=default.Table_de4427f05119413b8983f4e818072f98 [id=591b99acc8dd4e7388acb07fdf88a50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.132488 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e2fec8f4e14036af714427481f2f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.134841 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.136070 23647 tablet_service.cc:1505] Processing CreateTablet for tablet c3d145a33b964a4ca1f0001ac6cf990b (DEFAULT_TABLE table=default.Table_7ab2fc1342c440f9a0c55b4d52c748f8 [id=f59bbf36d431443a85154ff155cb0e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.136869 23783 ts_tablet_manager.cc:1916] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:21.137293 23790 tablet_bootstrap.cc:492] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:21.137471 23790 ts_tablet_manager.cc:1403] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.238s user 0.017s sys 0.026s
W20251023 20:15:21.140311 23605 leader_election.cc:343] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:21.141721 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26a680ad3aed487593bd1b63004dbbfc consensus state has no leader
I20251023 20:15:21.143556 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 5161336598884742ad403fc88c9d9f0b (DEFAULT_TABLE table=default.Table_e1d99e08ee354f119f078584381958e6 [id=a34d678a3e1f41d8b65903bc2126e4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.143867 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d145a33b964a4ca1f0001ac6cf990b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.144894 23790 raft_consensus.cc:359] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:21.145184 23790 raft_consensus.cc:385] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.145538 23790 raft_consensus.cc:740] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.145779 23790 consensus_queue.cc:260] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:21.147933 23790 ts_tablet_manager.cc:1434] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:21.149608 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5161336598884742ad403fc88c9d9f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.149881 23790 tablet_bootstrap.cc:492] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:21.151151 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9" } modify_external_catalogs: true
I20251023 20:15:21.151206 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b220ceff0924b26ae8a54dfb8705456. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.151351 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9" } modify_external_catalogs: true
I20251023 20:15:21.151774 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_a5331f0f356d4ed88cf6787eb94db083"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.152300 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5331f0f356d4ed88cf6787eb94db083 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.158129 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ae1a94b011412c851cd96946744fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.158383 23644 tablet_service.cc:1505] Processing CreateTablet for tablet 2202632a57514c849760509360113078 (DEFAULT_TABLE table=default.Table_b5c5f5ccad7d42a39151aa04d42db380 [id=de3be1729400426bb9957ccede88ceac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.164942 23638 tablet_service.cc:1505] Processing CreateTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 (DEFAULT_TABLE table=default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9 [id=d05789186efe404f94adbf2a543ba4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.167084 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2202632a57514c849760509360113078. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.168130 23605 leader_election.cc:304] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.168249 23942 raft_consensus.cc:493] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.167878 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "585a9af140d9479fb83d13e68a62300c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.166990 23634 tablet_service.cc:1505] Processing CreateTablet for tablet aa59141039de402e9ccce427604ecb31 (DEFAULT_TABLE table=default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.168179 23635 tablet_service.cc:1505] Processing CreateTablet for tablet 370a790a053446b2b06314c7db242270 (DEFAULT_TABLE table=default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.174333 23942 raft_consensus.cc:515] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.174598 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "364e3b9a07e44c8992f6d9cbda6a86e5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.176191 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.176433 23363 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e19f64a9ac44efba92928d0fa16819a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.178916 23638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.180694 23942 leader_election.cc:290] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.183122 23374 tablet_service.cc:1505] Processing CreateTablet for tablet e9e2fec8f4e14036af714427481f2f36 (DEFAULT_TABLE table=default.Table_de4427f05119413b8983f4e818072f98 [id=591b99acc8dd4e7388acb07fdf88a50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.184237 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e2fec8f4e14036af714427481f2f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.189203 23363 tablet_service.cc:1505] Processing CreateTablet for tablet a4d71f0cfdb6449fbd74b854c15326d6 (DEFAULT_TABLE table=default.Table_c67489b303534a86b98a072da3af3ec1 [id=d6f62018af4a4ed58edf91d90e430e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.189659 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d71f0cfdb6449fbd74b854c15326d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.191792 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
I20251023 20:15:21.193267 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3a893d487d5b46ca84e6d00ce6cf7137" } modify_external_catalogs: true
I20251023 20:15:21.193481 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3a893d487d5b46ca84e6d00ce6cf7137" } modify_external_catalogs: true
I20251023 20:15:21.196432 24171 raft_consensus.cc:493] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.198393 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8ac87d9615a74c5c858be1d6c31772a6" } modify_external_catalogs: true
I20251023 20:15:21.198565 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8ac87d9615a74c5c858be1d6c31772a6" } modify_external_catalogs: true
I20251023 20:15:21.197072 24171 raft_consensus.cc:515] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:21.201689 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet e418a62d00824cff89b37a7bace5c469
I20251023 20:15:21.165148 23785 rpcz_store.cc:276] 1023 20:15:19.752729 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.761714 (+ 8985us) service_pool.cc:225] Handling call
1023 20:15:20.977328 (+1215614us) ts_tablet_manager.cc:1151] Deleting tablet b90810781f724c78935a33d868ffa27d
1023 20:15:20.977337 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:20.988891 (+ 11554us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215415}
I20251023 20:15:21.205153 23790 tablet_bootstrap.cc:654] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.206570 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.208945 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:21.210443 23513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa0a46ffa626447eab4cdb956e798005. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.210371 23500 tablet_service.cc:1505] Processing CreateTablet for tablet aa59141039de402e9ccce427604ecb31 (DEFAULT_TABLE table=default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.211067 23501 tablet_service.cc:1505] Processing CreateTablet for tablet cd548fd9de284377ab557bd98f621068 (DEFAULT_TABLE table=default.Table_48c1e728bef74781a6d7353e1b1181be [id=cd774355ec3e41c29e00778e6fd3c23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.211395 23502 tablet_service.cc:1505] Processing CreateTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c (DEFAULT_TABLE table=default.Table_2496fe6d77e44734bfced35dee7d6596 [id=24a5b1c93ab84b2984243fc1a7f41e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.211735 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8da82e6d24b14a129b91bb5298139d07" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.212437 23535 raft_consensus.cc:3060] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:21.217628 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_df496168975b432da80cd2dd6c7edaf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.218293 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 8d22aa40e5844d78a8679a3ad1b08da9 (DEFAULT_TABLE table=default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 [id=79163b61e42a43d8879399cd5f31ce44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.219628 23498 tablet_service.cc:1505] Processing CreateTablet for tablet 19da816a483a4d5b9dc768ef697d54ec (DEFAULT_TABLE table=default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.219702 24160 raft_consensus.cc:2749] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.219908 24160 raft_consensus.cc:493] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.220098 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd548fd9de284377ab557bd98f621068. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.220042 24160 raft_consensus.cc:515] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:21.220098 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df496168975b432da80cd2dd6c7edaf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.220665 24160 leader_election.cc:290] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.220829 24160 raft_consensus.cc:493] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.220965 24160 raft_consensus.cc:515] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.221477 24160 leader_election.cc:290] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.223297 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa59141039de402e9ccce427604ecb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.225128 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3befdfc85bca42bb816e11d1dd832c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.226234 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d145a33b964a4ca1f0001ac6cf990b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.230234 23498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19da816a483a4d5b9dc768ef697d54ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.231681 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_316f315bbf2a4332a958a5f72ec606c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.231931 23786 tablet_bootstrap.cc:492] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
W20251023 20:15:21.232208 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_316f315bbf2a4332a958a5f72ec606c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.233528 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7824bcf5ff034be78ed7dee87e4cd78d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.233973 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7824bcf5ff034be78ed7dee87e4cd78d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.234788 23512 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.236131 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d22aa40e5844d78a8679a3ad1b08da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.238943 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370a790a053446b2b06314c7db242270. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.239288 23511 tablet_service.cc:1505] Processing CreateTablet for tablet a1efa4fcaeb049b484ca59b12025ce9a (DEFAULT_TABLE table=default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4 [id=abf30d62a0a84e239798cc1fca69fea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.239630 23510 tablet_service.cc:1505] Processing CreateTablet for tablet ea9be2a866cf46cfbec8db23a6d3f572 (DEFAULT_TABLE table=default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48 [id=a8c797d0049e4d1c9123db93f7ea12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.239718 23786 ts_tablet_manager.cc:1403] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.283s user 0.015s sys 0.007s
I20251023 20:15:21.239996 23512 tablet_service.cc:1505] Processing CreateTablet for tablet 9a4f012469fa4e8e827e485287ad76dd (DEFAULT_TABLE table=default.Table_e6e2be420a6342bc9d1311bd4fe6080c [id=8eeb69b1cebc42dcaae9f3306aa18013]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.245186 23369 tablet_service.cc:1505] Processing CreateTablet for tablet c3d145a33b964a4ca1f0001ac6cf990b (DEFAULT_TABLE table=default.Table_7ab2fc1342c440f9a0c55b4d52c748f8 [id=f59bbf36d431443a85154ff155cb0e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.249330 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_1b20af8a893a4715affda818cffeca13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.219568 23636 tablet_service.cc:1505] Processing CreateTablet for tablet fa0a46ffa626447eab4cdb956e798005 (DEFAULT_TABLE table=default.Table_934c5152009843bcbf11132bbe4c8ed2 [id=09f8335a5b03409c90559c8313f5d135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.249789 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e2fec8f4e14036af714427481f2f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.254809 23635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370a790a053446b2b06314c7db242270. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.255410 23636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa0a46ffa626447eab4cdb956e798005. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.221870 23631 tablet_service.cc:1505] Processing CreateTablet for tablet 8d22aa40e5844d78a8679a3ad1b08da9 (DEFAULT_TABLE table=default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 [id=79163b61e42a43d8879399cd5f31ce44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.257517 23500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa59141039de402e9ccce427604ecb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.257884 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9be2a866cf46cfbec8db23a6d3f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.258720 23512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a4f012469fa4e8e827e485287ad76dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.260948 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2202632a57514c849760509360113078. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.264132 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5161336598884742ad403fc88c9d9f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.267581 23787 tablet_bootstrap.cc:492] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:21.189396 23632 tablet_service.cc:1505] Processing CreateTablet for tablet cd548fd9de284377ab557bd98f621068 (DEFAULT_TABLE table=default.Table_48c1e728bef74781a6d7353e1b1181be [id=cd774355ec3e41c29e00778e6fd3c23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.224325 23633 tablet_service.cc:1505] Processing CreateTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c (DEFAULT_TABLE table=default.Table_2496fe6d77e44734bfced35dee7d6596 [id=24a5b1c93ab84b2984243fc1a7f41e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.269538 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d145a33b964a4ca1f0001ac6cf990b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.269654 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 5161336598884742ad403fc88c9d9f0b (DEFAULT_TABLE table=default.Table_e1d99e08ee354f119f078584381958e6 [id=a34d678a3e1f41d8b65903bc2126e4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.270627 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5161336598884742ad403fc88c9d9f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.272583 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 19da816a483a4d5b9dc768ef697d54ec (DEFAULT_TABLE table=default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.281220 23631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d22aa40e5844d78a8679a3ad1b08da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.282330 23787 ts_tablet_manager.cc:1403] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.317s user 0.009s sys 0.009s
I20251023 20:15:21.282835 23632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd548fd9de284377ab557bd98f621068. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.283938 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 (DEFAULT_TABLE table=default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9 [id=d05789186efe404f94adbf2a543ba4b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.286484 23633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3befdfc85bca42bb816e11d1dd832c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.289592 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19da816a483a4d5b9dc768ef697d54ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.290498 23376 tablet_service.cc:1505] Processing CreateTablet for tablet fa0a46ffa626447eab4cdb956e798005 (DEFAULT_TABLE table=default.Table_934c5152009843bcbf11132bbe4c8ed2 [id=09f8335a5b03409c90559c8313f5d135]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.291186 23642 tablet_service.cc:1505] Processing CreateTablet for tablet a1efa4fcaeb049b484ca59b12025ce9a (DEFAULT_TABLE table=default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4 [id=abf30d62a0a84e239798cc1fca69fea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.291567 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1efa4fcaeb049b484ca59b12025ce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.291746 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.292577 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b20af8a893a4715affda818cffeca13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.292932 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dafb9db710f94c328186e3179bf0b319" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.293926 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8e6b9df594194069b682dc1ece624a9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.294134 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 357) took 1515 ms. Trace:
I20251023 20:15:21.294890 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_af01354e99f74972ac276e46643f9dba" } new_table_name: "default.Table_fc6b97c7acc94953be861d738e8f830a" modify_external_catalogs: true
I20251023 20:15:21.295878 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_15836e4048de43f79d9a1d06e0072c3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.296325 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15836e4048de43f79d9a1d06e0072c3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.297484 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b90b24a3a90b47129da870a81b32f822" } new_table_name: "default.Table_083a66aed4484a38961dd79edd4d1c2e" modify_external_catalogs: true
I20251023 20:15:21.299320 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_698f8853c4774ccdae2e37d28f90f368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.300154 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_698f8853c4774ccdae2e37d28f90f368 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.304371 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4d011a709bd146c587b7b4eba10c6daf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.304848 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d011a709bd146c587b7b4eba10c6daf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.304950 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a21f26a0638f42d682abb0448b2a53e5" } new_table_name: "default.Table_d16baf5ed29d4a058c3ef64122fc7309" modify_external_catalogs: true
I20251023 20:15:21.305104 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_b5c5f5ccad7d42a39151aa04d42db380" } new_table_name: "default.Table_d4d5c2fd468445729fb6a7987712bd39" modify_external_catalogs: true
W20251023 20:15:21.307226 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e6b9df594194069b682dc1ece624a9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.303254 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4f8238f0492b4364b731c0bf5e2bd53a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.309211 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f8238f0492b4364b731c0bf5e2bd53a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:21.323256 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa794b3bec4b4b3993c814ea7cd02ee2 consensus state has no leader
W20251023 20:15:21.323545 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e consensus state has no leader
I20251023 20:15:21.328969 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1efa4fcaeb049b484ca59b12025ce9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.331202 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2202632a57514c849760509360113078 consensus state has no leader
W20251023 20:15:21.331727 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6f3abcaa81c4646bc1432bcbd1d4fcf consensus state has no leader
I20251023 20:15:21.334473 23786 raft_consensus.cc:359] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.337967 23786 raft_consensus.cc:385] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.338083 23786 raft_consensus.cc:740] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.338325 23786 consensus_queue.cc:260] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.340281 23504 tablet_service.cc:1505] Processing CreateTablet for tablet b446550686024c3ca80f106112a2110f (DEFAULT_TABLE table=default.Table_7a801afdfd4b4207b0e87c57ee93e01b [id=e74214affb4040149bd7c4db38c87270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.340672 23784 ts_tablet_manager.cc:1929] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:21.340893 23784 log.cc:1199] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/7f766bfe7dd94352abcaf34a200d00bc
I20251023 20:15:21.341413 23784 ts_tablet_manager.cc:1950] T 7f766bfe7dd94352abcaf34a200d00bc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:21.345295 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 857da3642f0846ce92aaed0491e89644 (DEFAULT_TABLE table=default.Table_6311c71cac724f7f8ba71a660247f056 [id=0f5e9dcbdd7f41efab4ded041edafd7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.346369 23786 ts_tablet_manager.cc:1434] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.103s user 0.001s sys 0.001s
I20251023 20:15:21.349457 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 7f766bfe7dd94352abcaf34a200d00bc (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]) successfully deleted
W20251023 20:15:21.349979 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2fa6a838b1bd409ba2a4e2074576ee23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fa6a838b1bd409ba2a4e2074576ee23 already in progress: creating tablet
W20251023 20:15:21.350423 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet bb475e303d07491bbc19dc3f628bceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb475e303d07491bbc19dc3f628bceb7 already in progress: creating tablet
W20251023 20:15:21.350883 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
W20251023 20:15:21.351533 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c97e2e0f839a4ef389fe100b6a81301e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97e2e0f839a4ef389fe100b6a81301e already in progress: creating tablet
W20251023 20:15:21.351816 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 40e0301fb5974ec5b6e8291230a736c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40e0301fb5974ec5b6e8291230a736c2 already in progress: creating tablet
W20251023 20:15:21.352037 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 0858358aabff47a69dc77f4669b225c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0858358aabff47a69dc77f4669b225c5 already in progress: creating tablet
I20251023 20:15:21.354668 23786 tablet_bootstrap.cc:492] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:21.362586 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8da82e6d24b14a129b91bb5298139d07" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.363318 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.363524 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 2202632a57514c849760509360113078 (DEFAULT_TABLE table=default.Table_b5c5f5ccad7d42a39151aa04d42db380 [id=de3be1729400426bb9957ccede88ceac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.363622 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 60e8942deb7544a09b21c3d1c2a3f99f (DEFAULT_TABLE table=default.Table_d0548a52907a4b1984665988fa00b425 [id=de89b68b2c194fada173999f7a0af035]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.363755 23510 tablet_service.cc:1505] Processing CreateTablet for tablet cf38936bb12846bd81a2c79ffbd18ccb (DEFAULT_TABLE table=default.Table_3a96b5d4c58047638a88c97bb9529c22 [id=b1ab817cbd9e47ea93ac15403fbb35a9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.363996 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because the tablet was not found. No further retry: Not found: Tablet not found: 9a6a2d2b25824dbdafaf2a9cabb0f061
I20251023 20:15:21.364754 23378 tablet_service.cc:1505] Processing CreateTablet for tablet aa59141039de402e9ccce427604ecb31 (DEFAULT_TABLE table=default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.365109 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 370a790a053446b2b06314c7db242270 (DEFAULT_TABLE table=default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.365288 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.365383 24171 leader_election.cc:290] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:21.365650 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa0a46ffa626447eab4cdb956e798005. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.368315 23785 rpcz_store.cc:276] 1023 20:15:19.752203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.769227 (+ 17024us) service_pool.cc:225] Handling call
1023 20:15:21.217642 (+1448415us) ts_tablet_manager.cc:1151] Deleting tablet a21ec49194e24723b31fd9c86f1f85e4
1023 20:15:21.217965 (+ 323us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.267616 (+ 49651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1389345}
I20251023 20:15:21.373147 23647 tablet_service.cc:1505] Processing CreateTablet for tablet 9a4f012469fa4e8e827e485287ad76dd (DEFAULT_TABLE table=default.Table_e6e2be420a6342bc9d1311bd4fe6080c [id=8eeb69b1cebc42dcaae9f3306aa18013]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.373580 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a4f012469fa4e8e827e485287ad76dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.377550 23377 tablet_service.cc:1505] Processing CreateTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c (DEFAULT_TABLE table=default.Table_2496fe6d77e44734bfced35dee7d6596 [id=24a5b1c93ab84b2984243fc1a7f41e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.383249 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 358) took 1624 ms. Trace:
I20251023 20:15:21.383369 23785 rpcz_store.cc:276] 1023 20:15:19.752320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.769133 (+ 16813us) service_pool.cc:225] Handling call
1023 20:15:21.368612 (+1599479us) ts_tablet_manager.cc:1151] Deleting tablet 7301d6121c9d4c8dbc642e895aa4e9f6
1023 20:15:21.368622 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.377021 (+ 8399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548826}
I20251023 20:15:21.383874 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 366) took 1565 ms. Trace:
I20251023 20:15:21.383920 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.383965 23785 rpcz_store.cc:276] 1023 20:15:19.818330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.819819 (+ 1489us) service_pool.cc:225] Handling call
1023 20:15:21.383652 (+1563833us) ts_tablet_manager.cc:1151] Deleting tablet 7540354f8577482785e6341fa9022315
1023 20:15:21.383665 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.383821 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563599}
I20251023 20:15:21.384171 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.384253 23785 tablet_replica.cc:333] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:21.384423 23785 raft_consensus.cc:2243] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:21.384606 23785 raft_consensus.cc:2272] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:21.384575 23508 tablet_service.cc:1505] Processing CreateTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 (DEFAULT_TABLE table=default.Table_a0beeea080b946158748ce5a6db61212 [id=108a2f4bdbd943139aa407a6197c1afd]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.386689 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e418a62d00824cff89b37a7bace5c469 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e418a62d00824cff89b37a7bace5c469 already in progress: creating tablet
W20251023 20:15:21.386992 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d82c92e863b2470699cc822b8cb8c14d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d82c92e863b2470699cc822b8cb8c14d already in progress: creating tablet
I20251023 20:15:21.387676 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b446550686024c3ca80f106112a2110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.389400 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 370a790a053446b2b06314c7db242270. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.388468 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb7935c4cf64e2f802ca12c1f63317b (DEFAULT_TABLE table=default.Table_31442b53b2f748d0895b0014aa8ae96f [id=90d69f76e2324f5ea1ea4fee2b9042a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.390893 23648 tablet_service.cc:1505] Processing CreateTablet for tablet ea9be2a866cf46cfbec8db23a6d3f572 (DEFAULT_TABLE table=default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48 [id=a8c797d0049e4d1c9123db93f7ea12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.394340 23502 tablet_service.cc:1505] Processing CreateTablet for tablet 4bce63866d524067b87145b07347ffcf (DEFAULT_TABLE table=default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.394852 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 60e8942deb7544a09b21c3d1c2a3f99f (DEFAULT_TABLE table=default.Table_d0548a52907a4b1984665988fa00b425 [id=de89b68b2c194fada173999f7a0af035]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.362532 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215e21ca449c481d8716783a65105101" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.396204 23379 tablet_service.cc:1505] Processing CreateTablet for tablet cd548fd9de284377ab557bd98f621068 (DEFAULT_TABLE table=default.Table_48c1e728bef74781a6d7353e1b1181be [id=cd774355ec3e41c29e00778e6fd3c23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.398097 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf38936bb12846bd81a2c79ffbd18ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.399423 23785 ts_tablet_manager.cc:1916] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:21.399765 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 8d22aa40e5844d78a8679a3ad1b08da9 (DEFAULT_TABLE table=default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 [id=79163b61e42a43d8879399cd5f31ce44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.401014 23373 tablet_service.cc:1505] Processing CreateTablet for tablet a1efa4fcaeb049b484ca59b12025ce9a (DEFAULT_TABLE table=default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4 [id=abf30d62a0a84e239798cc1fca69fea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.402698 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a0beeea080b946158748ce5a6db61212" } modify_external_catalogs: true
I20251023 20:15:21.402910 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a0beeea080b946158748ce5a6db61212" } modify_external_catalogs: true
I20251023 20:15:21.403290 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bd2b109bfa4804a372965e60ba6e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.405400 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_42ca9b5fa333476f9d6a35fcb9d0249a" } modify_external_catalogs: true
I20251023 20:15:21.405596 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_42ca9b5fa333476f9d6a35fcb9d0249a" } modify_external_catalogs: true
I20251023 20:15:21.407804 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3befdfc85bca42bb816e11d1dd832c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.407981 23790 tablet_bootstrap.cc:492] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:21.409564 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_dcdc9f5d2db74554b90c3565380b6a4b" } new_table_name: "default.Table_59879b02c1b84af2891ea5c6ebffbca7" modify_external_catalogs: true
I20251023 20:15:21.409662 24171 raft_consensus.cc:493] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.409701 23787 raft_consensus.cc:359] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.411607 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb7935c4cf64e2f802ca12c1f63317b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.426204 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 994a555b24e4432a84601c54d422aa5d consensus state has no leader
I20251023 20:15:21.430423 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet f9bd2b109bfa4804a372965e60ba6e85
I20251023 20:15:21.431743 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 9ed0aa7e36e64843ad6b2ad4b415832a
I20251023 20:15:21.441069 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857da3642f0846ce92aaed0491e89644. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.442618 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_30b62bd8bf6147a7a5800584576db269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.443192 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30b62bd8bf6147a7a5800584576db269 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.446341 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_1fd972f3964245fab674e2cfc26a8210" } modify_external_catalogs: true
I20251023 20:15:21.446514 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_1fd972f3964245fab674e2cfc26a8210" } modify_external_catalogs: true
I20251023 20:15:21.450901 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2202632a57514c849760509360113078. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.451956 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8e6b9df594194069b682dc1ece624a9e" } modify_external_catalogs: true
I20251023 20:15:21.452205 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8e6b9df594194069b682dc1ece624a9e" } modify_external_catalogs: true
I20251023 20:15:21.454754 23370 tablet_service.cc:1505] Processing CreateTablet for tablet ea9be2a866cf46cfbec8db23a6d3f572 (DEFAULT_TABLE table=default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48 [id=a8c797d0049e4d1c9123db93f7ea12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.455365 23790 ts_tablet_manager.cc:1403] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.258s user 0.000s sys 0.006s
I20251023 20:15:21.455883 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_156963dcf5454475a0cae4d52eff17f2" } modify_external_catalogs: true
I20251023 20:15:21.456035 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_156963dcf5454475a0cae4d52eff17f2" } modify_external_catalogs: true
W20251023 20:15:21.457229 23604 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 185592us
W20251023 20:15:21.457898 23604 leader_election.cc:343] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.458262 23790 raft_consensus.cc:359] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.457250 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.459028 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dafb9db710f94c328186e3179bf0b319" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.459043 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215e21ca449c481d8716783a65105101" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.459383 23372 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.460058 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 6558265ff9e249c68123577958f1401a with delete_type TABLET_DATA_DELETED (replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.461608 24171 raft_consensus.cc:515] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:21.462112 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bce63866d524067b87145b07347ffcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.462211 24171 leader_election.cc:290] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:21.463187 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa59141039de402e9ccce427604ecb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.463733 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9be2a866cf46cfbec8db23a6d3f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.463769 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 857da3642f0846ce92aaed0491e89644 (DEFAULT_TABLE table=default.Table_6311c71cac724f7f8ba71a660247f056 [id=0f5e9dcbdd7f41efab4ded041edafd7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.464020 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e8942deb7544a09b21c3d1c2a3f99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.464089 23783 ts_tablet_manager.cc:1929] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:21.464227 23783 log.cc:1199] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e0f8d8184a68453abd2792ccc7740aa1
I20251023 20:15:21.464039 23787 raft_consensus.cc:385] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.466943 23790 raft_consensus.cc:385] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.466382 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d22aa40e5844d78a8679a3ad1b08da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.468433 23790 raft_consensus.cc:740] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.468546 23503 tablet_service.cc:1505] Processing CreateTablet for tablet db758a7a7a89410f8ac1da72048fbc56 (DEFAULT_TABLE table=default.Table_df496168975b432da80cd2dd6c7edaf0 [id=3aa025681bb946e38410ee95fd48bddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.468735 23790 consensus_queue.cc:260] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.469220 23512 tablet_service.cc:1505] Processing CreateTablet for tablet 225ce6aa52c54f76bd56247f01219dbc (DEFAULT_TABLE table=default.Table_7824bcf5ff034be78ed7dee87e4cd78d [id=44239d72fdcd48258921b9490c822137]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.469547 23790 ts_tablet_manager.cc:1434] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251023 20:15:21.469846 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.471169 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd548fd9de284377ab557bd98f621068. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.471606 23790 tablet_bootstrap.cc:492] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:21.471731 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea9be2a866cf46cfbec8db23a6d3f572. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.472750 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1efa4fcaeb049b484ca59b12025ce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.466270 23363 tablet_service.cc:1505] Processing CreateTablet for tablet 19da816a483a4d5b9dc768ef697d54ec (DEFAULT_TABLE table=default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.474200 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 9a4f012469fa4e8e827e485287ad76dd (DEFAULT_TABLE table=default.Table_e6e2be420a6342bc9d1311bd4fe6080c [id=8eeb69b1cebc42dcaae9f3306aa18013]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.478955 23333 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.479991 23787 raft_consensus.cc:740] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.480172 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a2273d5581784756b737f8c7c17ed3a9
I20251023 20:15:21.479212 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.480954 23368 tablet_service.cc:1505] Processing CreateTablet for tablet b446550686024c3ca80f106112a2110f (DEFAULT_TABLE table=default.Table_7a801afdfd4b4207b0e87c57ee93e01b [id=e74214affb4040149bd7c4db38c87270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.479609 23364 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb7935c4cf64e2f802ca12c1f63317b (DEFAULT_TABLE table=default.Table_31442b53b2f748d0895b0014aa8ae96f [id=90d69f76e2324f5ea1ea4fee2b9042a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.479823 23360 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.481536 23360 tablet_service.cc:1505] Processing CreateTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 (DEFAULT_TABLE table=default.Table_a0beeea080b946158748ce5a6db61212 [id=108a2f4bdbd943139aa407a6197c1afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.480432 23372 tablet_service.cc:1505] Processing CreateTablet for tablet 857da3642f0846ce92aaed0491e89644 (DEFAULT_TABLE table=default.Table_6311c71cac724f7f8ba71a660247f056 [id=0f5e9dcbdd7f41efab4ded041edafd7d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.482409 23333 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 585a9af140d9479fb83d13e68a62300c
I20251023 20:15:21.482918 23333 leader_election.cc:304] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.483700 24171 raft_consensus.cc:2749] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:21.483913 23604 leader_election.cc:343] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:21.484200 23604 leader_election.cc:343] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.484903 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.484890 23374 tablet_service.cc:1505] Processing CreateTablet for tablet cf38936bb12846bd81a2c79ffbd18ccb (DEFAULT_TABLE table=default.Table_3a96b5d4c58047638a88c97bb9529c22 [id=b1ab817cbd9e47ea93ac15403fbb35a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.485311 23643 tablet_service.cc:1505] Processing CreateTablet for tablet 4bce63866d524067b87145b07347ffcf (DEFAULT_TABLE table=default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.485426 23333 leader_election.cc:343] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 451955aee3aa4bcb82252bcefbbcaefe
W20251023 20:15:21.485903 23605 leader_election.cc:343] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:21.486007 23333 leader_election.cc:343] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 6558265ff9e249c68123577958f1401a
I20251023 20:15:21.486199 23605 leader_election.cc:304] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.486415 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:21.486590 23605 leader_election.cc:343] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.486704 23605 leader_election.cc:304] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.486756 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 4bce63866d524067b87145b07347ffcf (DEFAULT_TABLE table=default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.487036 23605 leader_election.cc:343] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.487143 23605 leader_election.cc:304] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.490753 23364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb7935c4cf64e2f802ca12c1f63317b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.491389 23942 raft_consensus.cc:2749] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.491452 23360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bd2b109bfa4804a372965e60ba6e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.492095 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b446550686024c3ca80f106112a2110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.492717 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857da3642f0846ce92aaed0491e89644. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.492765 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857da3642f0846ce92aaed0491e89644. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.492885 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb7935c4cf64e2f802ca12c1f63317b (DEFAULT_TABLE table=default.Table_31442b53b2f748d0895b0014aa8ae96f [id=90d69f76e2324f5ea1ea4fee2b9042a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.493605 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf38936bb12846bd81a2c79ffbd18ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.493602 23787 consensus_queue.cc:260] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.494278 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bce63866d524067b87145b07347ffcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.494676 24160 raft_consensus.cc:493] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.495779 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19da816a483a4d5b9dc768ef697d54ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.495966 23787 ts_tablet_manager.cc:1434] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.212s user 0.000s sys 0.003s
I20251023 20:15:21.498584 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a4f012469fa4e8e827e485287ad76dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.500440 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb7935c4cf64e2f802ca12c1f63317b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.505218 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e8942deb7544a09b21c3d1c2a3f99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.506318 23512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225ce6aa52c54f76bd56247f01219dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.507798 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db758a7a7a89410f8ac1da72048fbc56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.504582 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 60e8942deb7544a09b21c3d1c2a3f99f (DEFAULT_TABLE table=default.Table_d0548a52907a4b1984665988fa00b425 [id=de89b68b2c194fada173999f7a0af035]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.522809 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e8942deb7544a09b21c3d1c2a3f99f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.524859 23333 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:21.528123 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 98a6a3425a49403e99c8ade114166e46 (DEFAULT_TABLE table=default.Table_316f315bbf2a4332a958a5f72ec606c1 [id=b4bbb2644eaf438f82f912407adc031f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.544569 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.545364 23786 tablet_bootstrap.cc:654] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.491664 23645 tablet_service.cc:1505] Processing CreateTablet for tablet cf38936bb12846bd81a2c79ffbd18ccb (DEFAULT_TABLE table=default.Table_3a96b5d4c58047638a88c97bb9529c22 [id=b1ab817cbd9e47ea93ac15403fbb35a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.548563 24176 raft_consensus.cc:2749] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.549904 23942 raft_consensus.cc:493] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.549906 23783 ts_tablet_manager.cc:1950] T e0f8d8184a68453abd2792ccc7740aa1 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:21.550519 23942 raft_consensus.cc:515] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:21.550732 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because the tablet was not found. No further retry: Not found: Tablet not found: f9bd2b109bfa4804a372965e60ba6e85
I20251023 20:15:21.551744 23644 tablet_service.cc:1505] Processing CreateTablet for tablet b446550686024c3ca80f106112a2110f (DEFAULT_TABLE table=default.Table_7a801afdfd4b4207b0e87c57ee93e01b [id=e74214affb4040149bd7c4db38c87270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.551965 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bce63866d524067b87145b07347ffcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.552168 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b446550686024c3ca80f106112a2110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.551777 24160 raft_consensus.cc:515] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.552687 24174 raft_consensus.cc:2749] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.555056 23787 tablet_bootstrap.cc:492] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:21.555241 23783 tablet_replica.cc:333] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:21.555450 23783 raft_consensus.cc:2243] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:21.555616 23783 raft_consensus.cc:2272] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:21.556246 23783 ts_tablet_manager.cc:1916] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:21.556398 23633 tablet_service.cc:1505] Processing CreateTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 (DEFAULT_TABLE table=default.Table_a0beeea080b946158748ce5a6db61212 [id=108a2f4bdbd943139aa407a6197c1afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.556623 23641 tablet_service.cc:1505] Processing CreateTablet for tablet db758a7a7a89410f8ac1da72048fbc56 (DEFAULT_TABLE table=default.Table_df496168975b432da80cd2dd6c7edaf0 [id=3aa025681bb946e38410ee95fd48bddc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.556852 23135 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet c6f3abcaa81c4646bc1432bcbd1d4fcf consensus state has no leader
I20251023 20:15:21.557368 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98a6a3425a49403e99c8ade114166e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.557745 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf
I20251023 20:15:21.557785 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf
I20251023 20:15:21.559679 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf38936bb12846bd81a2c79ffbd18ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.559895 23633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bd2b109bfa4804a372965e60ba6e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.564121 23942 leader_election.cc:290] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:21.566196 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
W20251023 20:15:21.566684 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a21ec49194e24723b31fd9c86f1f85e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a21ec49194e24723b31fd9c86f1f85e4 already in progress: creating tablet
W20251023 20:15:21.566931 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7301d6121c9d4c8dbc642e895aa4e9f6 already in progress: creating tablet
W20251023 20:15:21.567117 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 7540354f8577482785e6341fa9022315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7540354f8577482785e6341fa9022315 already in progress: creating tablet
I20251023 20:15:21.569608 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_f706cbf159424d419faa383ad30ebf77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.570158 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f706cbf159424d419faa383ad30ebf77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.571997 23144 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_0febf0b2ad3345998c666d31603db1b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.572177 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.572535 23144 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0febf0b2ad3345998c666d31603db1b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:21.572717 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.574107 23142 master_service.cc:954] ReplaceTablet: received request to replace tablet f103bece2fa240c19c23ac51046c13ed from {username='slave'} at 127.0.0.1:40384
I20251023 20:15:21.576148 24160 leader_election.cc:290] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.576932 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db758a7a7a89410f8ac1da72048fbc56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.580133 23640 tablet_service.cc:1505] Processing CreateTablet for tablet 98a6a3425a49403e99c8ade114166e46 (DEFAULT_TABLE table=default.Table_316f315bbf2a4332a958a5f72ec606c1 [id=b4bbb2644eaf438f82f912407adc031f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.586385 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet a2273d5581784756b737f8c7c17ed3a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.586704 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.586963 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.587356 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 225ce6aa52c54f76bd56247f01219dbc (DEFAULT_TABLE table=default.Table_7824bcf5ff034be78ed7dee87e4cd78d [id=44239d72fdcd48258921b9490c822137]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.599284 23471 leader_election.cc:304] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:21.589283 23515 tablet_service.cc:1505] Processing CreateTablet for tablet a2c022d78cac4959984d608d64df851b (DEFAULT_TABLE table=default.Table_698f8853c4774ccdae2e37d28f90f368 [id=3e597b14c5d944838c32914dc1779d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.603075 23784 tablet_replica.cc:333] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:21.604949 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fbad98d3ab24cc5b7726a92fba04cf8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.611389 23787 tablet_bootstrap.cc:654] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.611631 23501 tablet_service.cc:1505] Processing CreateTablet for tablet b3150a8478c144d285e999d5c9270fe0 (DEFAULT_TABLE table=default.Table_4f8238f0492b4364b731c0bf5e2bd53a [id=2db1100447a44e7fb43e87cdf7247d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.616585 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcbacd8e23bc4b129757d9faf89ce5b4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.617033 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fbad98d3ab24cc5b7726a92fba04cf8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.617311 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf (DEFAULT_TABLE table=default.Table_8e6b9df594194069b682dc1ece624a9e [id=759c4ec2b0544418a144bb7d3c02b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.617712 23513 tablet_service.cc:1505] Processing CreateTablet for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d (DEFAULT_TABLE table=default.Table_1b20af8a893a4715affda818cffeca13 [id=201b60a519df46fcb92de45acff546cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.618304 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 6abfda15e6c242b588d4ed4401008156 (DEFAULT_TABLE table=default.Table_15836e4048de43f79d9a1d06e0072c3c [id=b9d64b8c30014c5abbb2312d1b8017fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.618675 23509 tablet_service.cc:1505] Processing CreateTablet for tablet d227b45bfcc64d76a51cff347e6e1486 (DEFAULT_TABLE table=default.Table_a5331f0f356d4ed88cf6787eb94db083 [id=7bf105aabdc944afac7c5d1e5afb824b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.619001 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c022d78cac4959984d608d64df851b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.619582 23784 raft_consensus.cc:2243] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:21.619786 23784 raft_consensus.cc:2272] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:21.620350 23784 ts_tablet_manager.cc:1916] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:21.620843 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d227b45bfcc64d76a51cff347e6e1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.621994 23513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.619020 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcbacd8e23bc4b129757d9faf89ce5b4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.622786 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6abfda15e6c242b588d4ed4401008156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.623777 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet e0f8d8184a68453abd2792ccc7740aa1 (table default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]) successfully deleted
I20251023 20:15:21.624914 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.625000 24171 raft_consensus.cc:493] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.625195 24171 raft_consensus.cc:515] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:21.625829 24171 leader_election.cc:290] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:21.625834 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 757d41e027ba461ea051b38a2d685407 (DEFAULT_TABLE table=default.Table_4d011a709bd146c587b7b4eba10c6daf [id=dd33d0beee524dbbb6f9c73b46ca7912]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.626168 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757d41e027ba461ea051b38a2d685407. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.630082 23790 tablet_bootstrap.cc:654] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:21.631587 23794 raft_consensus.cc:2749] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:21.612423 23471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 49505.3us
I20251023 20:15:21.634001 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3150a8478c144d285e999d5c9270fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.637315 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225ce6aa52c54f76bd56247f01219dbc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.640022 23605 leader_election.cc:343] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.642481 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "451955aee3aa4bcb82252bcefbbcaefe" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.644270 23533 raft_consensus.cc:2468] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:21.646081 23333 leader_election.cc:304] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:21.646267 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet f103bece2fa240c19c23ac51046c13ed
I20251023 20:15:21.646514 24171 raft_consensus.cc:2804] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:21.646615 24171 raft_consensus.cc:493] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:21.646711 24171 raft_consensus.cc:3060] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:21.646871 23142 catalog_manager.cc:6719] ReplaceTablet: tablet f103bece2fa240c19c23ac51046c13ed deleted and replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224
W20251023 20:15:21.648033 23604 leader_election.cc:343] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.651883 24171 raft_consensus.cc:515] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.652561 24171 leader_election.cc:290] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.654453 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6558265ff9e249c68123577958f1401a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.658083 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98a6a3425a49403e99c8ade114166e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.658304 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e19f64a9ac44efba92928d0fa16819a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.658497 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0e61de7a93d450d9ed79eef82d7e5e3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:21.658960 23333 leader_election.cc:343] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 8e19f64a9ac44efba92928d0fa16819a
W20251023 20:15:21.660888 23333 leader_election.cc:343] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.661001 23333 leader_election.cc:304] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.661509 24171 raft_consensus.cc:2749] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.663251 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "451955aee3aa4bcb82252bcefbbcaefe" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:21.663455 23536 raft_consensus.cc:3060] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:21.665073 23333 leader_election.cc:343] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.665283 23333 leader_election.cc:304] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:21.665745 23333 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.666181 23333 leader_election.cc:304] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.666074 24171 raft_consensus.cc:2749] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.666674 24171 raft_consensus.cc:2749] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:21.667877 23604 leader_election.cc:343] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.668042 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.674993 23500 tablet_service.cc:1505] Processing CreateTablet for tablet 5cdde6b251b14df4bf44142ce263b84a (DEFAULT_TABLE table=default.Table_30b62bd8bf6147a7a5800584576db269 [id=054cce203d7e4437a8a7aa22df487453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.677861 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.678299 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 58859629a2f14aacbdb5e2cd0edc27bc (DEFAULT_TABLE table=default.Table_f706cbf159424d419faa383ad30ebf77 [id=d399717713214f3dbc8839f2908ba34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.680936 23500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cdde6b251b14df4bf44142ce263b84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.682076 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.682502 23604 leader_election.cc:304] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.682922 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58859629a2f14aacbdb5e2cd0edc27bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.683000 23604 consensus_peers.cc:597] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:21.683718 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.679237 23635 tablet_service.cc:1505] Processing CreateTablet for tablet d227b45bfcc64d76a51cff347e6e1486 (DEFAULT_TABLE table=default.Table_a5331f0f356d4ed88cf6787eb94db083 [id=7bf105aabdc944afac7c5d1e5afb824b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.686789 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet a2273d5581784756b737f8c7c17ed3a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.705255 24171 raft_consensus.cc:493] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:21.705524 24171 raft_consensus.cc:515] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.706310 24171 leader_election.cc:290] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.710359 23785 ts_tablet_manager.cc:1929] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:21.710492 23785 log.cc:1199] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/1bcdec7c16c943079e20bb486aecb431
I20251023 20:15:21.710877 23785 ts_tablet_manager.cc:1950] T 1bcdec7c16c943079e20bb486aecb431 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:21.723044 24137 catalog_manager.cc:1269] Loaded cluster ID: 67ec0cebd86e4322972b698542d64fab
I20251023 20:15:21.724339 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_9cac41d47a584e59a3c3e6fd69f9ad79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.724936 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cac41d47a584e59a3c3e6fd69f9ad79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.725661 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.726289 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.726248 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_3a96b5d4c58047638a88c97bb9529c22" } new_table_name: "default.Table_000ee28be94d48c394515c756bfe16bf" modify_external_catalogs: true
I20251023 20:15:21.726796 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6179bdbf369e4b32be88ca151e71e50f" } new_table_name: "default.Table_ede581d1b2824c17933809e745e8ab07" modify_external_catalogs: true
I20251023 20:15:21.727957 23141 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d943501e515b41e6a00142359321f818" } new_table_name: "default.Table_acb6fd552fa646929598fb1d07e2bb12" modify_external_catalogs: true
I20251023 20:15:21.727919 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_fc71da06b58e453a9fd0589907bf8987" } new_table_name: "default.Table_365d67170172442493eec0ae3918d5ff" modify_external_catalogs: true
I20251023 20:15:21.730512 23362 tablet_service.cc:1505] Processing CreateTablet for tablet db758a7a7a89410f8ac1da72048fbc56 (DEFAULT_TABLE table=default.Table_df496168975b432da80cd2dd6c7edaf0 [id=3aa025681bb946e38410ee95fd48bddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.730625 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8dc2f44cc1fc43e4a9851a0587a74edb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.731011 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db758a7a7a89410f8ac1da72048fbc56. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.731086 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dc2f44cc1fc43e4a9851a0587a74edb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.731457 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4c838e5585b948b9ba710086a76a2f3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.731886 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c838e5585b948b9ba710086a76a2f3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.732007 23372 tablet_service.cc:1505] Processing CreateTablet for tablet 225ce6aa52c54f76bd56247f01219dbc (DEFAULT_TABLE table=default.Table_7824bcf5ff034be78ed7dee87e4cd78d [id=44239d72fdcd48258921b9490c822137]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.734747 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 98a6a3425a49403e99c8ade114166e46 (DEFAULT_TABLE table=default.Table_316f315bbf2a4332a958a5f72ec606c1 [id=b4bbb2644eaf438f82f912407adc031f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.735100 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225ce6aa52c54f76bd56247f01219dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.735747 23145 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:21.737231 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98a6a3425a49403e99c8ade114166e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.739706 24160 raft_consensus.cc:2749] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.742421 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_39ee8d8d66f84963a998e326a50daf8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.742810 24137 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: loading cluster ID for follower catalog manager: success
W20251023 20:15:21.742983 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39ee8d8d66f84963a998e326a50daf8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.743327 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.743564 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.744778 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_20937abab3a8462cb783c7a4f34a9600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.745486 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20937abab3a8462cb783c7a4f34a9600 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:21.746553 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 377de4ea0bd6489284fac6b0fab52c8c consensus state has no leader
I20251023 20:15:21.747350 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c67489b303534a86b98a072da3af3ec1" } new_table_name: "default.Table_9c3cb66b734f447e82a09d9e653ea072" modify_external_catalogs: true
W20251023 20:15:21.747570 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b680b1cbc103403b91a1104248ecbdcf consensus state has no leader
W20251023 20:15:21.748958 23141 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a04c65155ce048cb82b150cbe20be849 consensus state has no leader
I20251023 20:15:21.749009 23147 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c7fa864846ef4acbba390d7f493a8b58" } modify_external_catalogs: true
W20251023 20:15:21.740509 23605 leader_election.cc:343] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.749191 23147 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c7fa864846ef4acbba390d7f493a8b58" } modify_external_catalogs: true
I20251023 20:15:21.749413 23635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d227b45bfcc64d76a51cff347e6e1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.750250 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e2c91fae0bf148f198063a62b0426f63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.750802 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2c91fae0bf148f198063a62b0426f63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.752149 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f21169f4f8814e739bc80b279a060a39" } modify_external_catalogs: true
I20251023 20:15:21.752336 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f21169f4f8814e739bc80b279a060a39" } modify_external_catalogs: true
I20251023 20:15:21.762701 23642 tablet_service.cc:1505] Processing CreateTablet for tablet b3150a8478c144d285e999d5c9270fe0 (DEFAULT_TABLE table=default.Table_4f8238f0492b4364b731c0bf5e2bd53a [id=2db1100447a44e7fb43e87cdf7247d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.763267 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3150a8478c144d285e999d5c9270fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.764678 24160 raft_consensus.cc:493] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:21.766148 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4d71f0cfdb6449fbd74b854c15326d6 consensus state has no leader
I20251023 20:15:21.766956 24160 raft_consensus.cc:515] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.767606 24160 leader_election.cc:290] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:21.771596 23146 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6311c71cac724f7f8ba71a660247f056" } modify_external_catalogs: true
I20251023 20:15:21.771800 23146 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6311c71cac724f7f8ba71a660247f056" } modify_external_catalogs: true
I20251023 20:15:21.773126 23145 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f4de4bcad2f34502a1b63889f5b5e58a" } modify_external_catalogs: true
I20251023 20:15:21.773290 23145 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f4de4bcad2f34502a1b63889f5b5e58a" } modify_external_catalogs: true
I20251023 20:15:21.774494 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_410a6baf06a642b18af37bebddac662f" } modify_external_catalogs: true
I20251023 20:15:21.774664 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_410a6baf06a642b18af37bebddac662f" } modify_external_catalogs: true
I20251023 20:15:21.775848 23605 leader_election.cc:304] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:21.777789 23605 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 126317us
I20251023 20:15:21.792404 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a6fe726ef844829a7ffec9d4494637" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:21.793733 23638 tablet_service.cc:1505] Processing CreateTablet for tablet 6abfda15e6c242b588d4ed4401008156 (DEFAULT_TABLE table=default.Table_15836e4048de43f79d9a1d06e0072c3c [id=b9d64b8c30014c5abbb2312d1b8017fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.796638 23147 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a841ffdc6cdd432dac7afe2e2b53f224
I20251023 20:15:21.797328 23787 tablet_bootstrap.cc:492] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:21.797560 23787 ts_tablet_manager.cc:1403] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.250s user 0.002s sys 0.003s
I20251023 20:15:21.797910 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 40c659c273c449939d0dd708d39b393f
I20251023 20:15:21.798162 23942 raft_consensus.cc:2749] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:21.798362 23604 leader_election.cc:343] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.800679 23634 tablet_service.cc:1505] Processing CreateTablet for tablet a2c022d78cac4959984d608d64df851b (DEFAULT_TABLE table=default.Table_698f8853c4774ccdae2e37d28f90f368 [id=3e597b14c5d944838c32914dc1779d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.808106 23146 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 857da3642f0846ce92aaed0491e89644
I20251023 20:15:21.809710 23787 raft_consensus.cc:359] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.809909 23787 raft_consensus.cc:385] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:21.809988 23787 raft_consensus.cc:740] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.810184 23787 consensus_queue.cc:260] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.811102 23787 ts_tablet_manager.cc:1434] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
W20251023 20:15:21.811857 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf38936bb12846bd81a2c79ffbd18ccb consensus state has no leader
I20251023 20:15:21.813795 23783 ts_tablet_manager.cc:1929] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:21.814100 23783 log.cc:1199] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/8da82e6d24b14a129b91bb5298139d07
I20251023 20:15:21.814518 23783 ts_tablet_manager.cc:1950] T 8da82e6d24b14a129b91bb5298139d07 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:21.820394 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a8eaa2167f82475ca2f6a110aba288ec
I20251023 20:15:21.822767 24137 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: acquiring CA information for follower catalog manager: success
I20251023 20:15:21.824716 23638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6abfda15e6c242b588d4ed4401008156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.833422 23636 tablet_service.cc:1505] Processing CreateTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf (DEFAULT_TABLE table=default.Table_8e6b9df594194069b682dc1ece624a9e [id=759c4ec2b0544418a144bb7d3c02b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.833879 23636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.833941 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet e194d03a6aa946fbb2c79d91b196440b
I20251023 20:15:21.840378 23366 tablet_service.cc:1505] Processing CreateTablet for tablet d227b45bfcc64d76a51cff347e6e1486 (DEFAULT_TABLE table=default.Table_a5331f0f356d4ed88cf6787eb94db083 [id=7bf105aabdc944afac7c5d1e5afb824b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.842610 23645 tablet_service.cc:1505] Processing CreateTablet for tablet 757d41e027ba461ea051b38a2d685407 (DEFAULT_TABLE table=default.Table_4d011a709bd146c587b7b4eba10c6daf [id=dd33d0beee524dbbb6f9c73b46ca7912]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.843588 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d227b45bfcc64d76a51cff347e6e1486. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.846477 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a6fe726ef844829a7ffec9d4494637" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:21.863930 23633 tablet_service.cc:1505] Processing CreateTablet for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d (DEFAULT_TABLE table=default.Table_1b20af8a893a4715affda818cffeca13 [id=201b60a519df46fcb92de45acff546cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.865530 23787 tablet_bootstrap.cc:492] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:21.866128 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.866581 23647 tablet_service.cc:1505] Processing CreateTablet for tablet 5cdde6b251b14df4bf44142ce263b84a (DEFAULT_TABLE table=default.Table_30b62bd8bf6147a7a5800584576db269 [id=054cce203d7e4437a8a7aa22df487453]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:21.867538 23605 leader_election.cc:343] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:21.867812 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.868021 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.868822 24137 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:15:21.868873 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.869217 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 58859629a2f14aacbdb5e2cd0edc27bc (DEFAULT_TABLE table=default.Table_f706cbf159424d419faa383ad30ebf77 [id=d399717713214f3dbc8839f2908ba34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.889444 23645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757d41e027ba461ea051b38a2d685407. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.892139 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 223726us
I20251023 20:15:21.899150 23605 leader_election.cc:304] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:21.899609 23942 raft_consensus.cc:2749] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:21.905169 23790 tablet_bootstrap.cc:492] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:21.905346 23790 ts_tablet_manager.cc:1403] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.434s user 0.013s sys 0.013s
I20251023 20:15:21.908092 23790 raft_consensus.cc:359] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.908305 23790 raft_consensus.cc:385] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:21.909689 23333 leader_election.cc:343] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 451955aee3aa4bcb82252bcefbbcaefe
I20251023 20:15:21.910154 23790 raft_consensus.cc:740] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:21.910420 23790 consensus_queue.cc:260] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:21.911445 23790 ts_tablet_manager.cc:1434] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:21.920120 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_50c7ff318dc44477817d0dbf91e7f68e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.920764 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50c7ff318dc44477817d0dbf91e7f68e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.922693 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e851810007344c51aa7a54934d6ad579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.922852 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d8b0d8499bdf41119dca8a299254c8ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.923205 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e851810007344c51aa7a54934d6ad579 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:21.925065 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8b0d8499bdf41119dca8a299254c8ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.938521 23503 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.943529 23503 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.945742 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.945897 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.947863 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:21.948540 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (DEFAULT_TABLE table=default.Table_0febf0b2ad3345998c666d31603db1b1 [id=9dd3c55753fa4ef5af5f31eddb6691b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.951256 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ea656dd8eee44e5f89ad208c70cddc39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:21.951313 23145 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_50c7ff318dc44477817d0dbf91e7f68e" } new_table_name: "default.Table_39425bc1be794ee8b1283c048094c708" modify_external_catalogs: true
W20251023 20:15:21.952994 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea656dd8eee44e5f89ad208c70cddc39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.953358 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.953859 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.954921 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_654a302882be43cd867d1b1ad3dc8bce" } new_table_name: "default.Table_c4502b0740ee4a34a1091754a3c9adf3" modify_external_catalogs: true
I20251023 20:15:21.955976 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59598ea40b9e4e5da0ec36f35f1094d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.958072 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet a2273d5581784756b737f8c7c17ed3a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.958561 23370 tablet_service.cc:1505] Processing CreateTablet for tablet b3150a8478c144d285e999d5c9270fe0 (DEFAULT_TABLE table=default.Table_4f8238f0492b4364b731c0bf5e2bd53a [id=2db1100447a44e7fb43e87cdf7247d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.958946 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3150a8478c144d285e999d5c9270fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.961373 23500 tablet_service.cc:1505] Processing CreateTablet for tablet dd07383242b74637bac46609f56c7e4d (DEFAULT_TABLE table=default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b [id=905ffd6fc69a44c39b12a7efecc70fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.963150 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d82a467c784f46dab749a2bfaa0aedbe" } new_table_name: "default.Table_14cd5c04c2b84209a82106f3424a8bd0" modify_external_catalogs: true
I20251023 20:15:21.964722 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_8c22652456c84553954550e1018e83e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.965258 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c22652456c84553954550e1018e83e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.966588 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6b180ae1e48347588da29cf3bd272d3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.967067 23145 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88f1df552f93427e91ca19c8170d69bd has no consensus state
W20251023 20:15:21.967131 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b180ae1e48347588da29cf3bd272d3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:21.969892 23333 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3894dd4feeea4c16861a6c9520059474
I20251023 20:15:21.971052 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf (DEFAULT_TABLE table=default.Table_8e6b9df594194069b682dc1ece624a9e [id=759c4ec2b0544418a144bb7d3c02b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.971472 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.972033 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 6abfda15e6c242b588d4ed4401008156 (DEFAULT_TABLE table=default.Table_15836e4048de43f79d9a1d06e0072c3c [id=b9d64b8c30014c5abbb2312d1b8017fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.972456 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6abfda15e6c242b588d4ed4401008156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.974360 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_69f62f4882e04aa1a32092b258c6acaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:21.974907 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69f62f4882e04aa1a32092b258c6acaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:21.976122 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_bbd7b9cada174d2f8e03189510c6ea3d" } new_table_name: "default.Table_a3de203f695c4ab09cef39511958ed08" modify_external_catalogs: true
I20251023 20:15:21.976888 23536 raft_consensus.cc:2468] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:21.978580 23373 tablet_service.cc:1505] Processing CreateTablet for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d (DEFAULT_TABLE table=default.Table_1b20af8a893a4715affda818cffeca13 [id=201b60a519df46fcb92de45acff546cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.979004 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.979846 23500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd07383242b74637bac46609f56c7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.982537 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.982717 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.982888 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.983203 23370 tablet_service.cc:1505] Processing CreateTablet for tablet 5cdde6b251b14df4bf44142ce263b84a (DEFAULT_TABLE table=default.Table_30b62bd8bf6147a7a5800584576db269 [id=054cce203d7e4437a8a7aa22df487453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.983561 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cdde6b251b14df4bf44142ce263b84a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:21.983621 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82ae1a94b011412c851cd96946744fc4 consensus state has no leader
I20251023 20:15:21.978590 23379 tablet_service.cc:1505] Processing CreateTablet for tablet 757d41e027ba461ea051b38a2d685407 (DEFAULT_TABLE table=default.Table_4d011a709bd146c587b7b4eba10c6daf [id=dd33d0beee524dbbb6f9c73b46ca7912]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.983968 23143 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]). Expected version 4 got 0
I20251023 20:15:21.982009 23365 tablet_service.cc:1505] Processing CreateTablet for tablet a2c022d78cac4959984d608d64df851b (DEFAULT_TABLE table=default.Table_698f8853c4774ccdae2e37d28f90f368 [id=3e597b14c5d944838c32914dc1779d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.986593 23376 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:21.986963 23376 tablet_service.cc:1505] Processing CreateTablet for tablet 58859629a2f14aacbdb5e2cd0edc27bc (DEFAULT_TABLE table=default.Table_f706cbf159424d419faa383ad30ebf77 [id=d399717713214f3dbc8839f2908ba34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.988081 23379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757d41e027ba461ea051b38a2d685407. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.989988 23376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58859629a2f14aacbdb5e2cd0edc27bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.990543 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c022d78cac4959984d608d64df851b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:21.991223 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 367) took 2141 ms. Trace:
I20251023 20:15:21.991357 23785 rpcz_store.cc:276] 1023 20:15:19.849469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.852303 (+ 2834us) service_pool.cc:225] Handling call
1023 20:15:21.384177 (+1531874us) ts_tablet_manager.cc:1151] Deleting tablet 1bcdec7c16c943079e20bb486aecb431
1023 20:15:21.384186 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.482880 (+ 98694us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:21.710231 (+227351us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:21.991175 (+280944us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":279403,"tablet-delete.queue_time_us":1531575}
I20251023 20:15:21.991933 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 368) took 2118 ms. Trace:
I20251023 20:15:21.992064 23785 rpcz_store.cc:276] 1023 20:15:19.873957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:19.874321 (+ 364us) service_pool.cc:225] Handling call
1023 20:15:21.991768 (+2117447us) ts_tablet_manager.cc:1151] Deleting tablet c9bfbf0acdc84f2f8fe553694ee9897c
1023 20:15:21.991787 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.991894 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2117091}
I20251023 20:15:21.992432 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 385) took 1879 ms. Trace:
I20251023 20:15:21.992444 23631 tablet_service.cc:1505] Processing CreateTablet for tablet dd07383242b74637bac46609f56c7e4d (DEFAULT_TABLE table=default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b [id=905ffd6fc69a44c39b12a7efecc70fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:21.992527 23785 rpcz_store.cc:276] 1023 20:15:20.112467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.112989 (+ 522us) service_pool.cc:225] Handling call
1023 20:15:21.992308 (+1879319us) ts_tablet_manager.cc:1151] Deleting tablet 72408899f710470baaf6fdb268cb04c8
1023 20:15:21.992320 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.992402 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878848}
I20251023 20:15:21.992866 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.993376 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 386) took 1880 ms. Trace:
I20251023 20:15:21.993492 23785 rpcz_store.cc:276] 1023 20:15:20.112636 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.113125 (+ 489us) service_pool.cc:225] Handling call
1023 20:15:21.992712 (+1879587us) ts_tablet_manager.cc:1151] Deleting tablet 7645c36c983f438f8b1223256fd3ad2c
1023 20:15:21.992719 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.993335 (+ 616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879123}
I20251023 20:15:21.994378 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 384) took 1882 ms. Trace:
I20251023 20:15:21.994556 23785 rpcz_store.cc:276] 1023 20:15:20.112320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.112900 (+ 580us) service_pool.cc:225] Handling call
1023 20:15:21.994231 (+1881331us) ts_tablet_manager.cc:1151] Deleting tablet 48436f3d8f1443db990a2d10b668b70a
1023 20:15:21.994241 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.994337 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873857}
I20251023 20:15:21.995366 23785 tablet_replica.cc:333] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:21.995672 23785 raft_consensus.cc:2243] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:21.995949 23785 raft_consensus.cc:2272] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:21.996623 23785 ts_tablet_manager.cc:1916] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:21.998850 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:21.999123 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:22.000106 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39a62c97e0a3459e91c24c1c2f92389a consensus state has no leader
I20251023 20:15:22.002243 23633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.003052 23786 tablet_bootstrap.cc:492] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.003412 23786 ts_tablet_manager.cc:1403] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.657s user 0.031s sys 0.050s
I20251023 20:15:22.004915 23790 tablet_bootstrap.cc:492] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.004859 23786 raft_consensus.cc:359] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.005285 23786 raft_consensus.cc:385] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.005407 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.005447 23786 raft_consensus.cc:740] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.005622 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.005689 23372 tablet_service.cc:1505] Processing CreateTablet for tablet dd07383242b74637bac46609f56c7e4d (DEFAULT_TABLE table=default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b [id=905ffd6fc69a44c39b12a7efecc70fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.006023 23362 tablet_service.cc:1505] Processing CreateTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 (DEFAULT_TABLE table=default.Table_c7fa864846ef4acbba390d7f493a8b58 [id=071ca3495096421eb75fea3fb71890ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.005941 23786 consensus_queue.cc:260] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.006503 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.006788 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.007117 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (DEFAULT_TABLE table=default.Table_0febf0b2ad3345998c666d31603db1b1 [id=9dd3c55753fa4ef5af5f31eddb6691b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.007197 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (DEFAULT_TABLE table=default.Table_0febf0b2ad3345998c666d31603db1b1 [id=9dd3c55753fa4ef5af5f31eddb6691b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.007582 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c022d78cac4959984d608d64df851b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.007965 23790 tablet_bootstrap.cc:654] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.009209 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd07383242b74637bac46609f56c7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.009606 23361 tablet_service.cc:1505] Processing CreateTablet for tablet 2c40164e687545c5b2d2751baaee4571 (DEFAULT_TABLE table=default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 [id=6defcb8665844b1d958c71d4fc34263a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.009720 23360 tablet_service.cc:1505] Processing CreateTablet for tablet a7f9428277fb4ebf989fed7569dee687 (DEFAULT_TABLE table=default.Table_9cac41d47a584e59a3c3e6fd69f9ad79 [id=6b26d29083ab46a29f00f2552ef20892]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.009877 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59598ea40b9e4e5da0ec36f35f1094d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.010125 23366 tablet_service.cc:1505] Processing CreateTablet for tablet b4798f6eea5745ba8a0d4b8cb173cc5c (DEFAULT_TABLE table=default.Table_8dc2f44cc1fc43e4a9851a0587a74edb [id=3bf5870046244dcd817441dce750a1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.009606 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 31df5c2c40de4d88b6ab1567b84ce4ef (DEFAULT_TABLE table=default.Table_20937abab3a8462cb783c7a4f34a9600 [id=afc43a90036d4b9cb050f975473f2e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.010661 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a841ffdc6cdd432dac7afe2e2b53f224. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.010896 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c40164e687545c5b2d2751baaee4571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.011219 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4798f6eea5745ba8a0d4b8cb173cc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.012239 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.013240 23360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7f9428277fb4ebf989fed7569dee687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.013191 23378 tablet_service.cc:1505] Processing CreateTablet for tablet 3e7909dba52748bdb78e08fa4b8a2d47 (DEFAULT_TABLE table=default.Table_e2c91fae0bf148f198063a62b0426f63 [id=b4c7610c982345c4a6f90e2025b41b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.013705 23498 tablet_service.cc:1505] Processing CreateTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 (DEFAULT_TABLE table=default.Table_c7fa864846ef4acbba390d7f493a8b58 [id=071ca3495096421eb75fea3fb71890ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.013823 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31df5c2c40de4d88b6ab1567b84ce4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.013940 23377 tablet_service.cc:1505] Processing CreateTablet for tablet aec7c5ea622949aca6c3dbe0e5f8e81e (DEFAULT_TABLE table=default.Table_4c838e5585b948b9ba710086a76a2f3f [id=04d988798f7f4b56be2fe5ed680539d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.014271 23377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec7c5ea622949aca6c3dbe0e5f8e81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.015374 23631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd07383242b74637bac46609f56c7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.017406 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 2c40164e687545c5b2d2751baaee4571 (DEFAULT_TABLE table=default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 [id=6defcb8665844b1d958c71d4fc34263a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.017910 23516 tablet_service.cc:1505] Processing CreateTablet for tablet a7f9428277fb4ebf989fed7569dee687 (DEFAULT_TABLE table=default.Table_9cac41d47a584e59a3c3e6fd69f9ad79 [id=6b26d29083ab46a29f00f2552ef20892]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.019176 23512 tablet_service.cc:1505] Processing CreateTablet for tablet 31df5c2c40de4d88b6ab1567b84ce4ef (DEFAULT_TABLE table=default.Table_20937abab3a8462cb783c7a4f34a9600 [id=afc43a90036d4b9cb050f975473f2e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.020918 23364 tablet_service.cc:1505] Processing CreateTablet for tablet fbab01ecb27a426188689408079eaa01 (DEFAULT_TABLE table=default.Table_39ee8d8d66f84963a998e326a50daf8b [id=ffe92ab0c88846539f80f1fa6cfe7d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.023401 23378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e7909dba52748bdb78e08fa4b8a2d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.007982 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.025595 23498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a841ffdc6cdd432dac7afe2e2b53f224. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:22.027400 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ca01bea2bb146ca8797a9592a840a81 consensus state has no leader
I20251023 20:15:22.020630 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_37c825c402604e08a862951e693de880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:22.028209 23632 tablet_service.cc:1505] Processing CreateTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 (DEFAULT_TABLE table=default.Table_c7fa864846ef4acbba390d7f493a8b58 [id=071ca3495096421eb75fea3fb71890ad]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:22.028528 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37c825c402604e08a862951e693de880 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.030567 23784 ts_tablet_manager.cc:1929] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:22.031654 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_837a7dbe47ef4dd3ba76317c430a26b1" } modify_external_catalogs: true
I20251023 20:15:22.031867 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_837a7dbe47ef4dd3ba76317c430a26b1" } modify_external_catalogs: true
I20251023 20:15:22.031957 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_df496168975b432da80cd2dd6c7edaf0" } modify_external_catalogs: true
I20251023 20:15:22.032069 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59598ea40b9e4e5da0ec36f35f1094d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.032114 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_39ee8d8d66f84963a998e326a50daf8b" } modify_external_catalogs: true
I20251023 20:15:22.032155 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_df496168975b432da80cd2dd6c7edaf0" } modify_external_catalogs: true
I20251023 20:15:22.032263 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_39ee8d8d66f84963a998e326a50daf8b" } modify_external_catalogs: true
I20251023 20:15:22.032410 23632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a841ffdc6cdd432dac7afe2e2b53f224. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.033515 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_86168bc39ef14e358f5356df1fe2e346" } new_table_name: "default.Table_3f23d19f3e734d84b7c5a1e0a0d83726" modify_external_catalogs: true
I20251023 20:15:22.033620 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_21f405dddb454882a3ad586d2deae0dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.034123 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21f405dddb454882a3ad586d2deae0dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.034477 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 1bcdec7c16c943079e20bb486aecb431 (table default.Table_6f178deaeeb94c4da6a40568c635e0b5 [id=cc451577eb8b4ba7937b8dbe4e859bc8]) successfully deleted
W20251023 20:15:22.034731 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
W20251023 20:15:22.034952 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:22.035130 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 7645c36c983f438f8b1223256fd3ad2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7645c36c983f438f8b1223256fd3ad2c already in progress: creating tablet
W20251023 20:15:22.035315 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 48436f3d8f1443db990a2d10b668b70a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48436f3d8f1443db990a2d10b668b70a already in progress: creating tablet
I20251023 20:15:22.035565 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7a413e9be0f843bc97a50978fefdec72" } modify_external_catalogs: true
I20251023 20:15:22.035720 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7a413e9be0f843bc97a50978fefdec72" } modify_external_catalogs: true
I20251023 20:15:22.037211 23784 log.cc:1199] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a2273d5581784756b737f8c7c17ed3a9
I20251023 20:15:22.037648 23784 ts_tablet_manager.cc:1950] T a2273d5581784756b737f8c7c17ed3a9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:22.037891 23647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cdde6b251b14df4bf44142ce263b84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.038566 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 2c40164e687545c5b2d2751baaee4571 (DEFAULT_TABLE table=default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 [id=6defcb8665844b1d958c71d4fc34263a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.040158 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.042968 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7f9428277fb4ebf989fed7569dee687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.043349 23643 tablet_service.cc:1505] Processing CreateTablet for tablet 31df5c2c40de4d88b6ab1567b84ce4ef (DEFAULT_TABLE table=default.Table_20937abab3a8462cb783c7a4f34a9600 [id=afc43a90036d4b9cb050f975473f2e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.043469 24187 tablet.cc:1722] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:22.044317 23786 ts_tablet_manager.cc:1434] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20251023 20:15:22.046620 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a2273d5581784756b737f8c7c17ed3a9 (table default.Table_156963dcf5454475a0cae4d52eff17f2 [id=9ed392ddcba74cd4a629eb0b728171ae]) successfully deleted
I20251023 20:15:22.047065 23506 tablet_service.cc:1505] Processing CreateTablet for tablet fbab01ecb27a426188689408079eaa01 (DEFAULT_TABLE table=default.Table_39ee8d8d66f84963a998e326a50daf8b [id=ffe92ab0c88846539f80f1fa6cfe7d1a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:22.048275 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
I20251023 20:15:22.049386 23514 tablet_service.cc:1505] Processing CreateTablet for tablet aec7c5ea622949aca6c3dbe0e5f8e81e (DEFAULT_TABLE table=default.Table_4c838e5585b948b9ba710086a76a2f3f [id=04d988798f7f4b56be2fe5ed680539d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.049479 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3894dd4feeea4c16861a6c9520059474" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:22.051017 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet fbab01ecb27a426188689408079eaa01
I20251023 20:15:22.050992 23650 tablet_service.cc:1505] Processing CreateTablet for tablet a7f9428277fb4ebf989fed7569dee687 (DEFAULT_TABLE table=default.Table_9cac41d47a584e59a3c3e6fd69f9ad79 [id=6b26d29083ab46a29f00f2552ef20892]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:22.051992 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5af107f9a144edb9080b7135ab8aeb9 consensus state has no leader
I20251023 20:15:22.052346 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 451955aee3aa4bcb82252bcefbbcaefe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.053457 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c40164e687545c5b2d2751baaee4571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.053826 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet db758a7a7a89410f8ac1da72048fbc56
I20251023 20:15:22.054371 23786 tablet_bootstrap.cc:492] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.055150 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58859629a2f14aacbdb5e2cd0edc27bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.056106 23502 tablet_service.cc:1505] Processing CreateTablet for tablet b4798f6eea5745ba8a0d4b8cb173cc5c (DEFAULT_TABLE table=default.Table_8dc2f44cc1fc43e4a9851a0587a74edb [id=3bf5870046244dcd817441dce750a1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.056241 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 3e7909dba52748bdb78e08fa4b8a2d47 (DEFAULT_TABLE table=default.Table_e2c91fae0bf148f198063a62b0426f63 [id=b4c7610c982345c4a6f90e2025b41b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.056430 23649 tablet_service.cc:1505] Processing CreateTablet for tablet b4798f6eea5745ba8a0d4b8cb173cc5c (DEFAULT_TABLE table=default.Table_8dc2f44cc1fc43e4a9851a0587a74edb [id=3bf5870046244dcd817441dce750a1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.056627 23505 tablet_service.cc:1505] Processing CreateTablet for tablet 3e7909dba52748bdb78e08fa4b8a2d47 (DEFAULT_TABLE table=default.Table_e2c91fae0bf148f198063a62b0426f63 [id=b4c7610c982345c4a6f90e2025b41b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.065685 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6d6d11ad70eb49409020a15552192008" } modify_external_catalogs: true
I20251023 20:15:22.065900 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_6d6d11ad70eb49409020a15552192008" } modify_external_catalogs: true
I20251023 20:15:22.066771 23512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31df5c2c40de4d88b6ab1567b84ce4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.067138 23503 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bcdec7c16c943079e20bb486aecb431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.067360 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:22.069329 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf already in progress: creating tablet
I20251023 20:15:22.072503 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 432) took 1177 ms. Trace:
I20251023 20:15:22.072641 23783 rpcz_store.cc:276] 1023 20:15:20.870590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.871160 (+ 570us) service_pool.cc:225] Handling call
1023 20:15:21.555164 (+684004us) ts_tablet_manager.cc:1151] Deleting tablet 8da82e6d24b14a129b91bb5298139d07
1023 20:15:21.555173 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.674209 (+119036us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:21.813666 (+139457us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.048249 (+234583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":683795}
I20251023 20:15:22.073132 23783 tablet_replica.cc:333] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:22.073344 23783 raft_consensus.cc:2243] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.073544 23783 raft_consensus.cc:2272] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.074105 23783 ts_tablet_manager.cc:1916] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.074760 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 8da82e6d24b14a129b91bb5298139d07 (table default.Table_58731fce9c224ff0b59d4b241110e518 [id=9ec90b724d89464a90821dd52282f77c]) successfully deleted
I20251023 20:15:22.075114 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31df5c2c40de4d88b6ab1567b84ce4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.075452 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c40164e687545c5b2d2751baaee4571. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.079751 23644 tablet_service.cc:1505] Processing CreateTablet for tablet aec7c5ea622949aca6c3dbe0e5f8e81e (DEFAULT_TABLE table=default.Table_4c838e5585b948b9ba710086a76a2f3f [id=04d988798f7f4b56be2fe5ed680539d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.084849 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7f9428277fb4ebf989fed7569dee687. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.085259 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.085711 23639 tablet_service.cc:1505] Processing CreateTablet for tablet fbab01ecb27a426188689408079eaa01 (DEFAULT_TABLE table=default.Table_39ee8d8d66f84963a998e326a50daf8b [id=ffe92ab0c88846539f80f1fa6cfe7d1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.089202 23787 tablet_bootstrap.cc:654] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.094085 23648 tablet_service.cc:1505] Processing CreateTablet for tablet a9e5dc42c01a4d5e9661f399f793f68f (DEFAULT_TABLE table=default.Table_e851810007344c51aa7a54934d6ad579 [id=0ddefd3e67364bc8bf23971811893894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.094405 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 01fa5f9c700b458ea147373d78551573
I20251023 20:15:22.094691 23640 tablet_service.cc:1505] Processing CreateTablet for tablet 88f1df552f93427e91ca19c8170d69bd (DEFAULT_TABLE table=default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.096808 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.099228 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 419e41d7a37f4b888cb242b089c28790
I20251023 20:15:22.100152 23364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbab01ecb27a426188689408079eaa01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.101190 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 370a790a053446b2b06314c7db242270
I20251023 20:15:22.105384 23333 leader_election.cc:304] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:22.107376 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbab01ecb27a426188689408079eaa01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.103873 23536 raft_consensus.cc:3060] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:22.110666 23505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e7909dba52748bdb78e08fa4b8a2d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.113001 24186 raft_consensus.cc:2804] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:22.113215 24186 raft_consensus.cc:697] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:22.113547 24186 consensus_queue.cc:237] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:22.115427 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
I20251023 20:15:22.115423 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:22.115911 23133 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because the tablet was not found. No further retry: Not found: Tablet not found: a841ffdc6cdd432dac7afe2e2b53f224
I20251023 20:15:22.120399 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4798f6eea5745ba8a0d4b8cb173cc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.124410 23515 tablet_service.cc:1505] Processing CreateTablet for tablet a9e5dc42c01a4d5e9661f399f793f68f (DEFAULT_TABLE table=default.Table_e851810007344c51aa7a54934d6ad579 [id=0ddefd3e67364bc8bf23971811893894]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:22.125355 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 60976.3us
W20251023 20:15:22.126178 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
W20251023 20:15:22.127323 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a8eaa2167f82475ca2f6a110aba288ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8eaa2167f82475ca2f6a110aba288ec already in progress: creating tablet
W20251023 20:15:22.128038 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
I20251023 20:15:22.131225 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_6443f81020254f45a1b7fd93ad217e09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:22.131554 23785 ts_tablet_manager.cc:1929] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.131762 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 88f1df552f93427e91ca19c8170d69bd (DEFAULT_TABLE table=default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:22.131909 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6443f81020254f45a1b7fd93ad217e09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.132746 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec7c5ea622949aca6c3dbe0e5f8e81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.140220 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_515e6170d0e34340a74759ab0a056f72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.140868 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_515e6170d0e34340a74759ab0a056f72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.145882 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_52ef4d97bc1b42eaaa12d286a51edd8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.147641 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52ef4d97bc1b42eaaa12d286a51edd8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:22.149834 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
I20251023 20:15:22.151362 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.151736 23790 tablet_bootstrap.cc:492] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.152436 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f1df552f93427e91ca19c8170d69bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.154390 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.156697 23509 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.161666 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
W20251023 20:15:22.164580 23333 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.164827 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.168766 23140 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_52ef4d97bc1b42eaaa12d286a51edd8b" } new_table_name: "default.Table_44c4bece35c24b7b9eae08ab8b835c78" modify_external_catalogs: true
I20251023 20:15:22.170423 23784 tablet_replica.cc:333] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:22.172672 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 63bdf5c80f2542f2a982d1de157d848d (DEFAULT_TABLE table=default.Table_d8b0d8499bdf41119dca8a299254c8ce [id=ddeeb413ab5b4a0ab8e01349eb17f066]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.173853 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.174425 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:22.179085 23140 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 118be7aa6380469db091d2788472e62b has no consensus state
I20251023 20:15:22.183296 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e5dc42c01a4d5e9661f399f793f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.186460 23784 raft_consensus.cc:2243] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.186784 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 63bdf5c80f2542f2a982d1de157d848d (DEFAULT_TABLE table=default.Table_d8b0d8499bdf41119dca8a299254c8ce [id=ddeeb413ab5b4a0ab8e01349eb17f066]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.189769 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63bdf5c80f2542f2a982d1de157d848d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.186928 23504 tablet_service.cc:1505] Processing CreateTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 (DEFAULT_TABLE table=default.Table_8c22652456c84553954550e1018e83e5 [id=ce7dc9691f2a4eeca898b4bf501fae49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.190223 23363 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.188045 23509 tablet_service.cc:1505] Processing CreateTablet for tablet b30e8ca2c96f4063a7f0e8f324c78b12 (DEFAULT_TABLE table=default.Table_6b180ae1e48347588da29cf3bd272d3e [id=e145b28bbae940b0bed6cf3cabd00611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.190932 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_e3186a2966f1433aa2cb5c751b84aa03" } new_table_name: "default.Table_2bc741a605f74c0dbcc7aecc45137db4" modify_external_catalogs: true
I20251023 20:15:22.192307 23790 ts_tablet_manager.cc:1403] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.150s user 0.024s sys 0.016s
I20251023 20:15:22.193940 23785 log.cc:1199] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/7f766bfe7dd94352abcaf34a200d00bc
I20251023 20:15:22.194432 23784 raft_consensus.cc:2272] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.194813 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b (DEFAULT_TABLE table=default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b [id=58cb342d7d6142858153064d66e3712c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.194108 23790 raft_consensus.cc:359] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.195811 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453eea5cdd2e47f29577c61b40ee59c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.196750 23333 leader_election.cc:304] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:22.198092 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b (DEFAULT_TABLE table=default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b [id=58cb342d7d6142858153064d66e3712c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.200858 23784 ts_tablet_manager.cc:1916] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.201256 24186 raft_consensus.cc:2749] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:22.196789 23367 tablet_service.cc:1505] Processing CreateTablet for tablet a9e5dc42c01a4d5e9661f399f793f68f (DEFAULT_TABLE table=default.Table_e851810007344c51aa7a54934d6ad579 [id=0ddefd3e67364bc8bf23971811893894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.201261 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 (DEFAULT_TABLE table=default.Table_8c22652456c84553954550e1018e83e5 [id=ce7dc9691f2a4eeca898b4bf501fae49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.205691 23363 tablet_service.cc:1505] Processing CreateTablet for tablet b30e8ca2c96f4063a7f0e8f324c78b12 (DEFAULT_TABLE table=default.Table_6b180ae1e48347588da29cf3bd272d3e [id=e145b28bbae940b0bed6cf3cabd00611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.213874 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 88f1df552f93427e91ca19c8170d69bd (DEFAULT_TABLE table=default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.214628 23790 raft_consensus.cc:385] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.214748 23790 raft_consensus.cc:740] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.221742 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:22.222018 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 744d31fccbbe4fa59c56e24136e3b2b9 consensus state has no leader
I20251023 20:15:22.222187 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec7c5ea622949aca6c3dbe0e5f8e81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.222301 23790 consensus_queue.cc:260] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.222885 23790 ts_tablet_manager.cc:1434] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20251023 20:15:22.223417 23942 raft_consensus.cc:493] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:22.224190 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63bdf5c80f2542f2a982d1de157d848d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.225513 23785 ts_tablet_manager.cc:1950] T 7f766bfe7dd94352abcaf34a200d00bc P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.226500 23790 tablet_bootstrap.cc:492] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.227227 23787 tablet_bootstrap.cc:492] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:22.229573 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.235150 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 6ee1974eb5e343b7953dadda7d5799b1 (DEFAULT_TABLE table=default.Table_515e6170d0e34340a74759ab0a056f72 [id=95f5faaee2b947fea16c52c55657db4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.238795 23641 tablet_service.cc:1505] Processing CreateTablet for tablet 63bdf5c80f2542f2a982d1de157d848d (DEFAULT_TABLE table=default.Table_d8b0d8499bdf41119dca8a299254c8ce [id=ddeeb413ab5b4a0ab8e01349eb17f066]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.241240 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_7cf41fd1dc6f47e5982f11504a3adf74" } new_table_name: "default.Table_deb8b605c159407ab25c527a038c8d44" modify_external_catalogs: true
I20251023 20:15:22.241204 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_385b3d01f133412a978baad54da4880d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.241829 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_385b3d01f133412a978baad54da4880d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.242497 23144 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_4d011a709bd146c587b7b4eba10c6daf" } new_table_name: "default.Table_d78ebb55a1754b5bb096fab938774ce1" modify_external_catalogs: true
I20251023 20:15:22.243618 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_aba7529cde2c47e39145a97c70f643ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.244084 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aba7529cde2c47e39145a97c70f643ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.245234 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_ec09021e59f8464f81cb6ceda1f1c567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.245760 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec09021e59f8464f81cb6ceda1f1c567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.247714 23942 raft_consensus.cc:515] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.250432 23370 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.250793 23373 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.251227 23635 tablet_service.cc:1505] Processing CreateTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 (DEFAULT_TABLE table=default.Table_8c22652456c84553954550e1018e83e5 [id=ce7dc9691f2a4eeca898b4bf501fae49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.251850 23642 tablet_service.cc:1505] Processing CreateTablet for tablet b30e8ca2c96f4063a7f0e8f324c78b12 (DEFAULT_TABLE table=default.Table_6b180ae1e48347588da29cf3bd272d3e [id=e145b28bbae940b0bed6cf3cabd00611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.253289 23364 tablet_service.cc:1505] Processing CreateTablet for tablet 95a06da281194f95b126483314f6debd (DEFAULT_TABLE table=default.Table_69f62f4882e04aa1a32092b258c6acaa [id=ee35b2f67f5f4d9488b1c6484ca4343a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.254099 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30e8ca2c96f4063a7f0e8f324c78b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.254779 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 6ee1974eb5e343b7953dadda7d5799b1 (DEFAULT_TABLE table=default.Table_515e6170d0e34340a74759ab0a056f72 [id=95f5faaee2b947fea16c52c55657db4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.255977 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f1df552f93427e91ca19c8170d69bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.256448 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e5dc42c01a4d5e9661f399f793f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.257597 23373 tablet_service.cc:1505] Processing CreateTablet for tablet 8a45d9b3031b46d390e0a1010489c718 (DEFAULT_TABLE table=default.Table_ea656dd8eee44e5f89ad208c70cddc39 [id=96188ab2961f4b14b71df728d91aa4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.253476 23372 tablet_service.cc:1505] Processing CreateTablet for tablet 1e5720c96730425dafb550094cf936cd (DEFAULT_TABLE table=default.Table_37c825c402604e08a862951e693de880 [id=6fdeeacf0d0949838d6062e9330e4524]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.263705 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 63ef1bae87c54aa988dfe49f828fe4f8 (DEFAULT_TABLE table=default.Table_6443f81020254f45a1b7fd93ad217e09 [id=4d43a0eda13041689a414ecbadb85e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.266404 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453eea5cdd2e47f29577c61b40ee59c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.265043 23365 tablet_service.cc:1505] Processing CreateTablet for tablet 118be7aa6380469db091d2788472e62b (DEFAULT_TABLE table=default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.269373 23379 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.270197 23141 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_736b5fe2d49043f896fa85aeed517265" } modify_external_catalogs: true
I20251023 20:15:22.270318 23147 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_01e28a3f1691446384d30911e4894273" } new_table_name: "default.Table_d1b58c19a42b4416a91f2614ebc3ea6b" modify_external_catalogs: true
I20251023 20:15:22.270679 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_1f76833d42104cee91c1406f649fa028"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.271198 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f76833d42104cee91c1406f649fa028 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.272024 23145 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_3ad1b2da425540ccbe8b55d22f379b4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.272492 23145 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ad1b2da425540ccbe8b55d22f379b4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.272843 23139 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f5d04920d5d94f63a2f285d7d824d648" } modify_external_catalogs: true
I20251023 20:15:22.286180 23141 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_736b5fe2d49043f896fa85aeed517265" } modify_external_catalogs: true
I20251023 20:15:22.286473 23140 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_b589e93e3314462d8ef529985314e501"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.287056 23140 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b589e93e3314462d8ef529985314e501 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:22.290649 23144 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 757d41e027ba461ea051b38a2d685407 consensus state has no leader
I20251023 20:15:22.292565 23139 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_f5d04920d5d94f63a2f285d7d824d648" } modify_external_catalogs: true
W20251023 20:15:22.295095 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d951134f6773410386bc57f7f90310c3 consensus state has no leader
I20251023 20:15:22.298074 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c01f9e9030974e2e8b43da4e68ef0145" } modify_external_catalogs: true
I20251023 20:15:22.298246 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_c01f9e9030974e2e8b43da4e68ef0145" } modify_external_catalogs: true
I20251023 20:15:22.302561 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbab01ecb27a426188689408079eaa01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.303593 23638 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.304068 23638 tablet_service.cc:1505] Processing CreateTablet for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b (DEFAULT_TABLE table=default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b [id=58cb342d7d6142858153064d66e3712c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.304423 23787 ts_tablet_manager.cc:1403] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.373s user 0.001s sys 0.003s
I20251023 20:15:22.306501 23142 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8c497d9a1dcc41aa8466741225b4de85" } modify_external_catalogs: true
I20251023 20:15:22.306710 23142 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_8c497d9a1dcc41aa8466741225b4de85" } modify_external_catalogs: true
I20251023 20:15:22.308323 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.308759 23634 tablet_service.cc:1505] Processing CreateTablet for tablet 95a06da281194f95b126483314f6debd (DEFAULT_TABLE table=default.Table_69f62f4882e04aa1a32092b258c6acaa [id=ee35b2f67f5f4d9488b1c6484ca4343a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.314445 23787 raft_consensus.cc:359] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.324890 23143 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_677942097b3a441287c448bb7c584a4f" } modify_external_catalogs: true
I20251023 20:15:22.328272 23141 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e
I20251023 20:15:22.329640 23942 leader_election.cc:290] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:22.332681 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.333665 24186 raft_consensus.cc:493] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:22.334228 23147 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e76f2e26dcd4cdfbf72de899f5a1458 consensus state has no leader
I20251023 20:15:22.334470 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.337122 23370 tablet_service.cc:1505] Processing CreateTablet for tablet b3a216aeebe649f79b6d873e1c59df37 (DEFAULT_TABLE table=default.Table_21f405dddb454882a3ad586d2deae0dd [id=0357e034941f45d2b977a566dfee9346]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.337163 23365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118be7aa6380469db091d2788472e62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.337958 23139 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 05933cb9433d48d285b2db82b289bdc8
I20251023 20:15:22.338941 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d9f422575c7c4251879130501ab721fd
I20251023 20:15:22.341944 24186 raft_consensus.cc:515] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.343143 23636 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.345989 23142 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet a9c5b69df4c04365b1ac1457fe8300cc
I20251023 20:15:22.347070 23370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a216aeebe649f79b6d873e1c59df37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.350533 23142 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e850560a68bd47a1bfb2bff80151368a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:22.352885 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23225d31c2e647398aa58494c627507e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:22.352284 23142 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e850560a68bd47a1bfb2bff80151368a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.356865 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee1974eb5e343b7953dadda7d5799b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.357700 24186 leader_election.cc:290] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:22.358666 23376 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.359717 23376 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.359881 23364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a06da281194f95b126483314f6debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.327257 23143 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_677942097b3a441287c448bb7c584a4f" } modify_external_catalogs: true
I20251023 20:15:22.360739 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 387) took 2221 ms. Trace:
I20251023 20:15:22.361617 23785 rpcz_store.cc:276] 1023 20:15:20.112783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.113237 (+ 454us) service_pool.cc:225] Handling call
1023 20:15:21.995024 (+1881787us) ts_tablet_manager.cc:1151] Deleting tablet 7f766bfe7dd94352abcaf34a200d00bc
1023 20:15:21.995035 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:21.999855 (+ 4820us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.130457 (+130602us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.334379 (+203922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872939}
I20251023 20:15:22.363626 23148 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_3a5e9b90e86340ed80fdb4a8679777e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.364269 23148 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a5e9b90e86340ed80fdb4a8679777e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.365650 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_10bf526d2893408887984427efbaa1f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.366142 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10bf526d2893408887984427efbaa1f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.370317 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ef1bae87c54aa988dfe49f828fe4f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:22.370486 23333 leader_election.cc:343] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.370721 23360 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.374591 23143 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 364e3b9a07e44c8992f6d9cbda6a86e5
I20251023 20:15:22.374779 23378 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.378394 23503 tablet_service.cc:1505] Processing CreateTablet for tablet b3a216aeebe649f79b6d873e1c59df37 (DEFAULT_TABLE table=default.Table_21f405dddb454882a3ad586d2deae0dd [id=0357e034941f45d2b977a566dfee9346]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.378614 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 1e5720c96730425dafb550094cf936cd (DEFAULT_TABLE table=default.Table_37c825c402604e08a862951e693de880 [id=6fdeeacf0d0949838d6062e9330e4524]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.379305 23513 tablet_service.cc:1505] Processing CreateTablet for tablet 95a06da281194f95b126483314f6debd (DEFAULT_TABLE table=default.Table_69f62f4882e04aa1a32092b258c6acaa [id=ee35b2f67f5f4d9488b1c6484ca4343a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.380899 23509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30e8ca2c96f4063a7f0e8f324c78b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.381747 23786 tablet_bootstrap.cc:654] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.383409 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee1974eb5e343b7953dadda7d5799b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.383920 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 8a45d9b3031b46d390e0a1010489c718 (DEFAULT_TABLE table=default.Table_ea656dd8eee44e5f89ad208c70cddc39 [id=96188ab2961f4b14b71df728d91aa4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.387434 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e5720c96730425dafb550094cf936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.387877 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.389362 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a45d9b3031b46d390e0a1010489c718. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.382010 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 7f766bfe7dd94352abcaf34a200d00bc (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]) successfully deleted
I20251023 20:15:22.392532 23641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63bdf5c80f2542f2a982d1de157d848d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.393328 24187 tablet.cc:1717] T 040a44e272b7410c98c109fbceb19192 P ea6853efbbcc484eb7529c7e9b37a47a: Skipping requested alter to schema version 4, tablet already version 4
I20251023 20:15:22.394145 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ea63540149c4c02abb8ae51209060a3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:22.395118 23942 raft_consensus.cc:493] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:22.395931 23787 raft_consensus.cc:385] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.396027 23787 raft_consensus.cc:740] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.396236 23787 consensus_queue.cc:260] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.396632 23787 ts_tablet_manager.cc:1434] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.092s user 0.000s sys 0.001s
I20251023 20:15:22.397886 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 383) took 2256 ms. Trace:
I20251023 20:15:22.398694 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 364e3b9a07e44c8992f6d9cbda6a86e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.400321 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 (DEFAULT_TABLE table=default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a [id=273447c687354c109e50ecd2bd95084b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.400378 23785 rpcz_store.cc:276] 1023 20:15:20.111973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.112078 (+ 105us) service_pool.cc:225] Handling call
1023 20:15:22.362093 (+2250015us) ts_tablet_manager.cc:1151] Deleting tablet 46f700ea3db04a4bad539c943e850a44
1023 20:15:22.362102 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.368757 (+ 6655us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239726}
I20251023 20:15:22.400892 23787 tablet_bootstrap.cc:492] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:22.401198 23604 leader_election.cc:343] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.402973 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 388) took 2157 ms. Trace:
I20251023 20:15:22.403060 23785 rpcz_store.cc:276] 1023 20:15:20.245779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.249128 (+ 3349us) service_pool.cc:225] Handling call
1023 20:15:22.400647 (+2151519us) ts_tablet_manager.cc:1151] Deleting tablet ac55e14745534a02a2f6a422cab6b149
1023 20:15:22.400656 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.402857 (+ 2201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2143345}
I20251023 20:15:22.404031 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 405) took 1814 ms. Trace:
I20251023 20:15:22.404111 23785 rpcz_store.cc:276] 1023 20:15:20.589565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591514 (+ 1949us) service_pool.cc:225] Handling call
1023 20:15:22.403756 (+1812242us) ts_tablet_manager.cc:1151] Deleting tablet 6480cdc1afda4671a346044f232b083e
1023 20:15:22.403766 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.403850 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1799220}
I20251023 20:15:22.404502 23636 tablet_service.cc:1505] Processing CreateTablet for tablet 8a45d9b3031b46d390e0a1010489c718 (DEFAULT_TABLE table=default.Table_ea656dd8eee44e5f89ad208c70cddc39 [id=96188ab2961f4b14b71df728d91aa4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.404608 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 403) took 1815 ms. Trace:
I20251023 20:15:22.404693 23785 rpcz_store.cc:276] 1023 20:15:20.589361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591471 (+ 2110us) service_pool.cc:225] Handling call
1023 20:15:22.404326 (+1812855us) ts_tablet_manager.cc:1151] Deleting tablet 352812b56f8d4feb818a364d6adfd86d
1023 20:15:22.404334 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.404413 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1799678}
I20251023 20:15:22.404765 23631 tablet_service.cc:1505] Processing CreateTablet for tablet b3a216aeebe649f79b6d873e1c59df37 (DEFAULT_TABLE table=default.Table_21f405dddb454882a3ad586d2deae0dd [id=0357e034941f45d2b977a566dfee9346]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.405196 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 409) took 1815 ms. Trace:
I20251023 20:15:22.405275 23785 rpcz_store.cc:276] 1023 20:15:20.589912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591616 (+ 1704us) service_pool.cc:225] Handling call
1023 20:15:22.404900 (+1813284us) ts_tablet_manager.cc:1151] Deleting tablet fb01b3f4ff7f41e98063ebf76abeace6
1023 20:15:22.404908 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.404985 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800175}
I20251023 20:15:22.405700 23785 tablet_replica.cc:333] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.408752 23785 raft_consensus.cc:2243] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:22.409099 23785 pending_rounds.cc:70] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:22.409180 23785 pending_rounds.cc:77] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082138580217856 op_type: NO_OP noop_request { }
I20251023 20:15:22.409320 23785 raft_consensus.cc:2889] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:22.409432 23785 raft_consensus.cc:2272] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.412449 23785 ts_tablet_manager.cc:1916] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.423092 23785 ts_tablet_manager.cc:1929] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:22.423362 23785 log.cc:1199] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/451955aee3aa4bcb82252bcefbbcaefe
I20251023 20:15:22.423892 23785 ts_tablet_manager.cc:1950] T 451955aee3aa4bcb82252bcefbbcaefe P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.424392 23635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453eea5cdd2e47f29577c61b40ee59c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.424844 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 402) took 1835 ms. Trace:
I20251023 20:15:22.425009 23785 rpcz_store.cc:276] 1023 20:15:20.589227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591439 (+ 2212us) service_pool.cc:225] Handling call
1023 20:15:22.405478 (+1814039us) ts_tablet_manager.cc:1151] Deleting tablet 451955aee3aa4bcb82252bcefbbcaefe
1023 20:15:22.405486 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.419990 (+ 14504us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.422891 (+ 2901us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.424787 (+ 1896us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":330,"tablet-delete.queue_time_us":1800666}
I20251023 20:15:22.425999 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 407) took 1836 ms. Trace:
I20251023 20:15:22.426144 23785 rpcz_store.cc:276] 1023 20:15:20.589738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591557 (+ 1819us) service_pool.cc:225] Handling call
1023 20:15:22.425829 (+1834272us) ts_tablet_manager.cc:1151] Deleting tablet cec51c54ea534b1da4b1d170e79d9f1c
1023 20:15:22.425839 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.425945 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1820020}
I20251023 20:15:22.426206 23942 raft_consensus.cc:515] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.426715 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 408) took 1836 ms. Trace:
I20251023 20:15:22.426864 23942 leader_election.cc:290] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:22.426796 23785 rpcz_store.cc:276] 1023 20:15:20.589838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591580 (+ 1742us) service_pool.cc:225] Handling call
1023 20:15:22.426564 (+1834984us) ts_tablet_manager.cc:1151] Deleting tablet 951b2276d3324c2d999c8d7fb97767bc
1023 20:15:22.426574 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.426667 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1819974}
I20251023 20:15:22.427235 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 406) took 1837 ms. Trace:
I20251023 20:15:22.427721 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cff337a6e728489b99fa8c5d4dfece2b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:22.427809 23633 tablet_service.cc:1505] Processing CreateTablet for tablet 1e5720c96730425dafb550094cf936cd (DEFAULT_TABLE table=default.Table_37c825c402604e08a862951e693de880 [id=6fdeeacf0d0949838d6062e9330e4524]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.428109 23783 ts_tablet_manager.cc:1929] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.428295 23783 log.cc:1199] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/dafb9db710f94c328186e3179bf0b319
I20251023 20:15:22.428436 23785 rpcz_store.cc:276] 1023 20:15:20.589662 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591537 (+ 1875us) service_pool.cc:225] Handling call
1023 20:15:22.427091 (+1835554us) ts_tablet_manager.cc:1151] Deleting tablet d9f73477a06542c0b215ae86716cdc4e
1023 20:15:22.427100 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.427180 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1819690}
I20251023 20:15:22.429109 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 404) took 1839 ms. Trace:
I20251023 20:15:22.429260 23785 rpcz_store.cc:276] 1023 20:15:20.589458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.591493 (+ 2035us) service_pool.cc:225] Handling call
1023 20:15:22.428865 (+1837372us) ts_tablet_manager.cc:1151] Deleting tablet 514cb3b1ed024041ac160a7d9acff618
1023 20:15:22.428873 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.429037 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1821362}
I20251023 20:15:22.429723 23785 tablet_replica.cc:333] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.429956 23785 raft_consensus.cc:2243] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.430192 23785 raft_consensus.cc:2272] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.430790 23785 ts_tablet_manager.cc:1916] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.434585 23631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a216aeebe649f79b6d873e1c59df37. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:22.437176 23604 leader_election.cc:343] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.437575 23785 ts_tablet_manager.cc:1929] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.437755 23785 log.cc:1199] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e31aebfc87734ffeb2c0b25c07cb3f70
I20251023 20:15:22.438169 23785 ts_tablet_manager.cc:1950] T e31aebfc87734ffeb2c0b25c07cb3f70 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.439051 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 401) took 1856 ms. Trace:
I20251023 20:15:22.439224 23785 rpcz_store.cc:276] 1023 20:15:20.582123 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.582457 (+ 334us) service_pool.cc:225] Handling call
1023 20:15:22.429643 (+1847186us) ts_tablet_manager.cc:1151] Deleting tablet e31aebfc87734ffeb2c0b25c07cb3f70
1023 20:15:22.429651 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.434382 (+ 4731us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.437351 (+ 2969us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.439008 (+ 1657us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815215}
I20251023 20:15:22.439921 23785 tablet_replica.cc:333] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.440178 23785 raft_consensus.cc:2243] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.440394 23785 raft_consensus.cc:2272] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.441002 23785 ts_tablet_manager.cc:1916] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.446903 23783 ts_tablet_manager.cc:1950] T dafb9db710f94c328186e3179bf0b319 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:22.447274 23785 ts_tablet_manager.cc:1929] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.447469 23785 log.cc:1199] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/585a9af140d9479fb83d13e68a62300c
I20251023 20:15:22.447713 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 433) took 1576 ms. Trace:
I20251023 20:15:22.447926 23785 ts_tablet_manager.cc:1950] T 585a9af140d9479fb83d13e68a62300c P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.447840 23783 rpcz_store.cc:276] 1023 20:15:20.870730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.871407 (+ 677us) service_pool.cc:225] Handling call
1023 20:15:22.073002 (+1201595us) ts_tablet_manager.cc:1151] Deleting tablet dafb9db710f94c328186e3179bf0b319
1023 20:15:22.073016 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.222145 (+149129us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.413288 (+191143us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.447674 (+ 34386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1201422}
I20251023 20:15:22.448287 23783 tablet_replica.cc:333] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:22.448889 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 410) took 1747 ms. Trace:
I20251023 20:15:22.449035 23785 rpcz_store.cc:276] 1023 20:15:20.701531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.702936 (+ 1405us) service_pool.cc:225] Handling call
1023 20:15:22.439849 (+1736913us) ts_tablet_manager.cc:1151] Deleting tablet 585a9af140d9479fb83d13e68a62300c
1023 20:15:22.439858 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.444301 (+ 4443us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.447168 (+ 2867us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.448837 (+ 1669us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1736613}
I20251023 20:15:22.449589 23785 tablet_replica.cc:333] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.449626 23643 tablet_service.cc:1505] Processing CreateTablet for tablet 6ee1974eb5e343b7953dadda7d5799b1 (DEFAULT_TABLE table=default.Table_515e6170d0e34340a74759ab0a056f72 [id=95f5faaee2b947fea16c52c55657db4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.449808 23785 raft_consensus.cc:2243] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.449986 23785 raft_consensus.cc:2272] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.450551 23785 ts_tablet_manager.cc:1916] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.450548 23507 tablet_service.cc:1505] Processing CreateTablet for tablet 63ef1bae87c54aa988dfe49f828fe4f8 (DEFAULT_TABLE table=default.Table_6443f81020254f45a1b7fd93ad217e09 [id=4d43a0eda13041689a414ecbadb85e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.456394 23636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a45d9b3031b46d390e0a1010489c718. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.457289 23785 ts_tablet_manager.cc:1929] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.457485 23785 log.cc:1199] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:22.457578 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e5dc42c01a4d5e9661f399f793f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.457923 23785 ts_tablet_manager.cc:1950] T d0e61de7a93d450d9ed79eef82d7e5e3 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.458781 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 411) took 1661 ms. Trace:
I20251023 20:15:22.458952 23785 rpcz_store.cc:276] 1023 20:15:20.797479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.798917 (+ 1438us) service_pool.cc:225] Handling call
1023 20:15:22.449502 (+1650585us) ts_tablet_manager.cc:1151] Deleting tablet d0e61de7a93d450d9ed79eef82d7e5e3
1023 20:15:22.449511 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.453671 (+ 4160us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.457171 (+ 3500us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.458740 (+ 1569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638853}
I20251023 20:15:22.459314 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee1974eb5e343b7953dadda7d5799b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.459631 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 412) took 1558 ms. Trace:
I20251023 20:15:22.459792 23785 rpcz_store.cc:276] 1023 20:15:20.901632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.901696 (+ 64us) service_pool.cc:225] Handling call
1023 20:15:22.459475 (+1557779us) ts_tablet_manager.cc:1151] Deleting tablet a86a25659ec44d7c825783036342ed45
1023 20:15:22.459484 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.459600 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51,"tablet-delete.queue_time_us":1557588}
I20251023 20:15:22.460392 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 413) took 1557 ms. Trace:
I20251023 20:15:22.460557 23785 rpcz_store.cc:276] 1023 20:15:20.902554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.902596 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:22.460272 (+1557676us) ts_tablet_manager.cc:1151] Deleting tablet 79e3d422a66f432789c0cf4a00e57af2
1023 20:15:22.460282 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.460367 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557430}
I20251023 20:15:22.461131 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 414) took 1557 ms. Trace:
I20251023 20:15:22.461285 23785 rpcz_store.cc:276] 1023 20:15:20.903152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.903191 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:22.460961 (+1557770us) ts_tablet_manager.cc:1151] Deleting tablet 2fa6a838b1bd409ba2a4e2074576ee23
1023 20:15:22.460970 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.461065 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557610}
I20251023 20:15:22.461779 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 425) took 1482 ms. Trace:
I20251023 20:15:22.461932 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b30e8ca2c96f4063a7f0e8f324c78b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.461935 23785 rpcz_store.cc:276] 1023 20:15:20.978954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.978996 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:22.461668 (+1482672us) ts_tablet_manager.cc:1151] Deleting tablet ca11ddd5d97544cf916b87fd3dff219d
1023 20:15:22.461676 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.461752 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482274}
I20251023 20:15:22.462442 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 426) took 1479 ms. Trace:
I20251023 20:15:22.462749 23785 rpcz_store.cc:276] 1023 20:15:20.983410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.983466 (+ 56us) service_pool.cc:225] Handling call
1023 20:15:22.462286 (+1478820us) ts_tablet_manager.cc:1151] Deleting tablet c97e2e0f839a4ef389fe100b6a81301e
1023 20:15:22.462295 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.462407 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1475659}
I20251023 20:15:22.463263 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 427) took 1475 ms. Trace:
I20251023 20:15:22.463428 23785 rpcz_store.cc:276] 1023 20:15:20.988246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.988296 (+ 50us) service_pool.cc:225] Handling call
1023 20:15:22.463121 (+1474825us) ts_tablet_manager.cc:1151] Deleting tablet bb475e303d07491bbc19dc3f628bceb7
1023 20:15:22.463129 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.463226 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473991}
I20251023 20:15:22.463960 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 432) took 1290 ms. Trace:
I20251023 20:15:22.464129 23785 rpcz_store.cc:276] 1023 20:15:21.173957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.174071 (+ 114us) service_pool.cc:225] Handling call
1023 20:15:22.463818 (+1289747us) ts_tablet_manager.cc:1151] Deleting tablet 40e0301fb5974ec5b6e8291230a736c2
1023 20:15:22.463826 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.463924 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287490}
I20251023 20:15:22.464581 23785 tablet_replica.cc:333] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.464793 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a06da281194f95b126483314f6debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.464849 23785 raft_consensus.cc:2243] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.465133 23785 raft_consensus.cc:2272] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.465775 23785 ts_tablet_manager.cc:1916] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.466236 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e7909dba52748bdb78e08fa4b8a2d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.467643 23506 tablet_service.cc:1505] Processing CreateTablet for tablet 118be7aa6380469db091d2788472e62b (DEFAULT_TABLE table=default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.468824 23638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.470204 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a45d9b3031b46d390e0a1010489c718. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.471732 23787 tablet_bootstrap.cc:654] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.471979 23785 ts_tablet_manager.cc:1929] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.472167 23785 log.cc:1199] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/8e19f64a9ac44efba92928d0fa16819a
I20251023 20:15:22.472836 23785 ts_tablet_manager.cc:1950] T 8e19f64a9ac44efba92928d0fa16819a P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.473641 23633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e5720c96730425dafb550094cf936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.473739 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 431) took 1302 ms. Trace:
I20251023 20:15:22.473873 23785 rpcz_store.cc:276] 1023 20:15:21.171596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.171712 (+ 116us) service_pool.cc:225] Handling call
1023 20:15:22.464504 (+1292792us) ts_tablet_manager.cc:1151] Deleting tablet 8e19f64a9ac44efba92928d0fa16819a
1023 20:15:22.464512 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.468922 (+ 4410us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.471871 (+ 2949us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.473682 (+ 1811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287875}
I20251023 20:15:22.474546 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 447) took 1072 ms. Trace:
I20251023 20:15:22.474701 23785 rpcz_store.cc:276] 1023 20:15:21.408218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.408266 (+ 48us) service_pool.cc:225] Handling call
1023 20:15:22.474374 (+1066108us) ts_tablet_manager.cc:1151] Deleting tablet 0858358aabff47a69dc77f4669b225c5
1023 20:15:22.474384 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.474484 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014862}
I20251023 20:15:22.475181 23785 tablet_replica.cc:333] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.476485 23503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a216aeebe649f79b6d873e1c59df37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.476771 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8317a54a5f7b4afe8da9d336268efe43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.477094 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.480351 23498 tablet_service.cc:1505] Processing CreateTablet for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8 (DEFAULT_TABLE table=default.Table_10bf526d2893408887984427efbaa1f2 [id=933b756e69bf4d6d96590acd69f45267]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.480796 23498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.481247 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.484087 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e5720c96730425dafb550094cf936cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.475407 23785 raft_consensus.cc:2243] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.485381 23785 raft_consensus.cc:2272] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.486159 23785 ts_tablet_manager.cc:1916] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.486320 23513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a06da281194f95b126483314f6debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.489495 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 660d84eb63a4490886aba1777d449ee2 (DEFAULT_TABLE table=default.Table_ec09021e59f8464f81cb6ceda1f1c567 [id=2fd818d424a44efdab6cf6fda7537ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.489586 23512 tablet_service.cc:1505] Processing CreateTablet for tablet e3df1fd83e3c4ac29a7846736701044a (DEFAULT_TABLE table=default.Table_3ad1b2da425540ccbe8b55d22f379b4a [id=7810e4d5193c4559891f1603fe361e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.490058 23512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3df1fd83e3c4ac29a7846736701044a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.491914 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660d84eb63a4490886aba1777d449ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.493265 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118be7aa6380469db091d2788472e62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.493404 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 255521d943d34713aaf773b15f9168c6 (DEFAULT_TABLE table=default.Table_1f76833d42104cee91c1406f649fa028 [id=a48bcc6da6504ccb941c18dd5e289c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.493762 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255521d943d34713aaf773b15f9168c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.498502 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 9749c6f3fe5545ab93bf49aae97e9818 (DEFAULT_TABLE table=default.Table_b589e93e3314462d8ef529985314e501 [id=bdeb476ba31d425893262ab98d5b579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.498978 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9749c6f3fe5545ab93bf49aae97e9818. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.504766 23507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ef1bae87c54aa988dfe49f828fe4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.510615 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f1df552f93427e91ca19c8170d69bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.518229 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4798f6eea5745ba8a0d4b8cb173cc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.521554 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 364e3b9a07e44c8992f6d9cbda6a86e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.524333 23785 ts_tablet_manager.cc:1929] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.525025 23785 log.cc:1199] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/6558265ff9e249c68123577958f1401a
I20251023 20:15:22.525671 23785 ts_tablet_manager.cc:1950] T 6558265ff9e249c68123577958f1401a P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.526458 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 448) took 1097 ms. Trace:
I20251023 20:15:22.526654 23785 rpcz_store.cc:276] 1023 20:15:21.429256 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.435104 (+ 5848us) service_pool.cc:225] Handling call
1023 20:15:22.475093 (+1039989us) ts_tablet_manager.cc:1151] Deleting tablet 6558265ff9e249c68123577958f1401a
1023 20:15:22.475102 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.518930 (+ 43828us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.524206 (+ 5276us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.526410 (+ 2204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014912}
I20251023 20:15:22.527374 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 455) took 1050 ms. Trace:
I20251023 20:15:22.527570 23785 rpcz_store.cc:276] 1023 20:15:21.477207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.479122 (+ 1915us) service_pool.cc:225] Handling call
1023 20:15:22.527221 (+1048099us) ts_tablet_manager.cc:1151] Deleting tablet e418a62d00824cff89b37a7bace5c469
1023 20:15:22.527230 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.527333 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046538}
I20251023 20:15:22.528112 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 457) took 1050 ms. Trace:
I20251023 20:15:22.528277 23785 rpcz_store.cc:276] 1023 20:15:21.477379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.479744 (+ 2365us) service_pool.cc:225] Handling call
1023 20:15:22.527978 (+1048234us) ts_tablet_manager.cc:1151] Deleting tablet d82c92e863b2470699cc822b8cb8c14d
1023 20:15:22.527986 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.528084 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046700}
I20251023 20:15:22.528795 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 452) took 1078 ms. Trace:
I20251023 20:15:22.528952 23785 rpcz_store.cc:276] 1023 20:15:21.476820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.486321 (+ 9501us) service_pool.cc:225] Handling call
1023 20:15:22.528672 (+1042351us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:22.528679 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.528758 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042136}
I20251023 20:15:22.529409 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 463) took 1006 ms. Trace:
I20251023 20:15:22.529603 23785 rpcz_store.cc:276] 1023 20:15:21.522745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.739523 (+216778us) service_pool.cc:225] Handling call
1023 20:15:22.529269 (+789746us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:22.529277 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.529372 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":785813}
I20251023 20:15:22.530119 23785 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47314 (request call id 465) took 1006 ms. Trace:
I20251023 20:15:22.530282 23785 rpcz_store.cc:276] 1023 20:15:21.523698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.743490 (+219792us) service_pool.cc:225] Handling call
1023 20:15:22.529986 (+786496us) ts_tablet_manager.cc:1151] Deleting tablet 9ed0aa7e36e64843ad6b2ad4b415832a
1023 20:15:22.529995 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.530077 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":786332}
I20251023 20:15:22.530794 23785 tablet_replica.cc:333] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.531068 23785 raft_consensus.cc:2243] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.531298 23785 raft_consensus.cc:2272] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.531992 23785 ts_tablet_manager.cc:1916] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.535456 23790 tablet_bootstrap.cc:654] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.539711 23785 ts_tablet_manager.cc:1929] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:22.539960 23785 log.cc:1199] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a2273d5581784756b737f8c7c17ed3a9
I20251023 20:15:22.540392 23785 ts_tablet_manager.cc:1950] T a2273d5581784756b737f8c7c17ed3a9 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.542299 23785 tablet_replica.cc:333] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.542577 23785 raft_consensus.cc:2243] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.542834 23785 raft_consensus.cc:2272] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.542964 23142 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_273b73dbda4e41aea636fdc95149120a" } new_table_name: "default.Table_8cda343a0b0c499b9d4190bb861443a7" modify_external_catalogs: true
I20251023 20:15:22.543500 23785 ts_tablet_manager.cc:1916] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.549582 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_4fb0e33f75594ca59d2ecb391873d804"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.550148 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fb0e33f75594ca59d2ecb391873d804 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.552076 23785 ts_tablet_manager.cc:1929] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.552281 23785 log.cc:1199] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/3894dd4feeea4c16861a6c9520059474
I20251023 20:15:22.552839 23785 ts_tablet_manager.cc:1950] T 3894dd4feeea4c16861a6c9520059474 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.555644 23785 tablet_replica.cc:333] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.555893 23785 raft_consensus.cc:2243] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.556205 23785 raft_consensus.cc:2272] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.556793 23785 ts_tablet_manager.cc:1916] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.560164 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 (DEFAULT_TABLE table=default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a [id=273447c687354c109e50ecd2bd95084b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.560571 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8317a54a5f7b4afe8da9d336268efe43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.561607 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_d8b0d8499bdf41119dca8a299254c8ce" } new_table_name: "default.Table_676121013e244d349484cd6c7cb75a8f" modify_external_catalogs: true
I20251023 20:15:22.564576 23785 ts_tablet_manager.cc:1929] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.564764 23785 log.cc:1199] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/23225d31c2e647398aa58494c627507e
I20251023 20:15:22.565177 23785 ts_tablet_manager.cc:1950] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.566341 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.567521 23785 tablet_replica.cc:333] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.567826 23785 raft_consensus.cc:2243] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.568070 23785 raft_consensus.cc:2272] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.568198 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 2e735945647148478201eeffa1d03632 (DEFAULT_TABLE table=default.Table_aba7529cde2c47e39145a97c70f643ed [id=c949af1c17974ec991f6bb58c01c4f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.568661 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e735945647148478201eeffa1d03632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.568919 23785 ts_tablet_manager.cc:1916] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:22.568976 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63bdf5c80f2542f2a982d1de157d848d consensus state has no leader
W20251023 20:15:22.570701 23132 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 88f1df552f93427e91ca19c8170d69bd consensus state has no leader
I20251023 20:15:22.571765 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_527cb9b18f414f43872b4f9dd4c433e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.573098 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_527cb9b18f414f43872b4f9dd4c433e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.577013 23143 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a6f1c53dfe964d538a77e7b6570d9a9a" } new_table_name: "default.Table_d42255cb02014b30a183176f660d20f3" modify_external_catalogs: true
I20251023 20:15:22.582722 23139 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_1ec08f875c4f4a438cd2081db4326eb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.583592 23139 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ec08f875c4f4a438cd2081db4326eb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:22.584632 23143 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efad9b4a8e2e446f917df7ca26fbf789 consensus state has no leader
I20251023 20:15:22.586653 23143 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_e6177be8581a457cb9113fcc941d6c65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.587265 23143 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6177be8581a457cb9113fcc941d6c65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.587896 23790 tablet_bootstrap.cc:492] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.588068 23790 ts_tablet_manager.cc:1403] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.365s user 0.000s sys 0.010s
I20251023 20:15:22.589133 23790 raft_consensus.cc:359] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.589314 23790 raft_consensus.cc:385] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.589485 23790 raft_consensus.cc:740] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.589746 23790 consensus_queue.cc:260] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.590454 23148 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_a87b53598d1a4c0b9324ab792449a67d" } new_table_name: "default.Table_bce343d5f0fc48b5aef18a2668354047" modify_external_catalogs: true
I20251023 20:15:22.590420 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.590935 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.592072 23790 ts_tablet_manager.cc:1434] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251023 20:15:22.592355 23790 tablet_bootstrap.cc:492] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.570757 23516 tablet_service.cc:1505] Processing CreateTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 (DEFAULT_TABLE table=default.Table_385b3d01f133412a978baad54da4880d [id=1d6f12be6ba342c98a64503cd58d8ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.593979 23502 tablet_service.cc:1505] Processing CreateTablet for tablet a0079496170849c6b3c233e1cd78e40e (DEFAULT_TABLE table=default.Table_3a5e9b90e86340ed80fdb4a8679777e6 [id=22d88af65ebd43389f6e58573143816c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.596194 23790 tablet_bootstrap.cc:654] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.598817 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0cbd6bec674cf5831486f500cc7b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.601630 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 (DEFAULT_TABLE table=default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.602438 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0079496170849c6b3c233e1cd78e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.611589 23790 tablet_bootstrap.cc:492] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.611773 23790 ts_tablet_manager.cc:1403] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251023 20:15:22.612025 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f2a4f1d52984af58d6286671389b6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.612598 23790 raft_consensus.cc:359] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.612807 23790 raft_consensus.cc:385] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.612890 23790 raft_consensus.cc:740] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.613180 23790 consensus_queue.cc:260] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.608706 23785 ts_tablet_manager.cc:1929] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.619114 23785 log.cc:1199] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e0f8d8184a68453abd2792ccc7740aa1
I20251023 20:15:22.619472 23785 ts_tablet_manager.cc:1950] T e0f8d8184a68453abd2792ccc7740aa1 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.619993 24184 raft_consensus.cc:493] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:22.621261 23333 leader_election.cc:343] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 23225d31c2e647398aa58494c627507e
I20251023 20:15:22.621402 23333 leader_election.cc:304] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:22.621613 23333 raft_consensus.cc:2686] T 23225d31c2e647398aa58494c627507e P d220aba7c9c0400398ffde917376ee48: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251023 20:15:22.620151 24184 raft_consensus.cc:515] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.624711 23785 tablet_replica.cc:333] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:22.624909 23785 raft_consensus.cc:2243] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.625102 23785 raft_consensus.cc:2272] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.627280 23790 ts_tablet_manager.cc:1434] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251023 20:15:22.627766 23785 ts_tablet_manager.cc:1916] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.629009 24184 leader_election.cc:290] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:22.629523 23790 tablet_bootstrap.cc:492] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:22.629771 23471 leader_election.cc:343] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 1bcdec7c16c943079e20bb486aecb431
I20251023 20:15:22.633795 23790 tablet_bootstrap.cc:654] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.640482 23785 ts_tablet_manager.cc:1929] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:22.648577 23785 log.cc:1199] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/364e3b9a07e44c8992f6d9cbda6a86e5
I20251023 20:15:22.649117 23785 ts_tablet_manager.cc:1950] T 364e3b9a07e44c8992f6d9cbda6a86e5 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:22.650111 23786 tablet_bootstrap.cc:492] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.657156 23787 tablet_bootstrap.cc:492] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:22.658833 23784 ts_tablet_manager.cc:1929] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.659365 23784 log.cc:1199] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/451955aee3aa4bcb82252bcefbbcaefe
I20251023 20:15:22.659237 23786 ts_tablet_manager.cc:1403] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.605s user 0.003s sys 0.014s
I20251023 20:15:22.662487 23146 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_20216f4bd8cd4150bfb55aa1e918e7c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.663157 23146 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20216f4bd8cd4150bfb55aa1e918e7c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.663301 23145 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a" } modify_external_catalogs: true
I20251023 20:15:22.663463 23145 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a" } modify_external_catalogs: true
I20251023 20:15:22.664661 23140 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_385b3d01f133412a978baad54da4880d" } modify_external_catalogs: true
I20251023 20:15:22.664812 23139 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_21f405dddb454882a3ad586d2deae0dd" } new_table_name: "default.Table_4d8885fad7e54f0eae2d0720f3062ec0" modify_external_catalogs: true
I20251023 20:15:22.664907 23144 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_083a66aed4484a38961dd79edd4d1c2e" } modify_external_catalogs: true
I20251023 20:15:22.665050 23144 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_083a66aed4484a38961dd79edd4d1c2e" } modify_external_catalogs: true
I20251023 20:15:22.665222 23141 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_5e48fffc45314e45a6f7ac8bba5ab31d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.665738 23141 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e48fffc45314e45a6f7ac8bba5ab31d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:22.667060 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 46f700ea3db04a4bad539c943e850a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f700ea3db04a4bad539c943e850a44 already in progress: creating tablet
W20251023 20:15:22.667368 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet ac55e14745534a02a2f6a422cab6b149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac55e14745534a02a2f6a422cab6b149 already in progress: creating tablet
W20251023 20:15:22.667584 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
W20251023 20:15:22.667788 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 352812b56f8d4feb818a364d6adfd86d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352812b56f8d4feb818a364d6adfd86d already in progress: creating tablet
W20251023 20:15:22.667992 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
I20251023 20:15:22.668254 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 451955aee3aa4bcb82252bcefbbcaefe (table default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
W20251023 20:15:22.668464 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet cec51c54ea534b1da4b1d170e79d9f1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cec51c54ea534b1da4b1d170e79d9f1c already in progress: creating tablet
W20251023 20:15:22.668828 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 951b2276d3324c2d999c8d7fb97767bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 951b2276d3324c2d999c8d7fb97767bc already in progress: creating tablet
W20251023 20:15:22.669708 23605 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 419246us
I20251023 20:15:22.670776 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet dafb9db710f94c328186e3179bf0b319 (table default.Table_82f6a78855d1488b9a8085c3bfbebab3 [id=f033b532f86d452c8f14e7f40ead8d62]) successfully deleted
I20251023 20:15:22.671615 23783 raft_consensus.cc:2243] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.671808 23783 raft_consensus.cc:2272] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.672453 23783 ts_tablet_manager.cc:1916] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.672729 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ea63540149c4c02abb8ae51209060a3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:22.673189 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.673729 23632 tablet_service.cc:1505] Processing CreateTablet for tablet 255521d943d34713aaf773b15f9168c6 (DEFAULT_TABLE table=default.Table_1f76833d42104cee91c1406f649fa028 [id=a48bcc6da6504ccb941c18dd5e289c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.673729 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cff337a6e728489b99fa8c5d4dfece2b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:22.674099 23632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255521d943d34713aaf773b15f9168c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.664877 23140 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_385b3d01f133412a978baad54da4880d" } modify_external_catalogs: true
I20251023 20:15:22.677407 23633 tablet_service.cc:1505] Processing CreateTablet for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8 (DEFAULT_TABLE table=default.Table_10bf526d2893408887984427efbaa1f2 [id=933b756e69bf4d6d96590acd69f45267]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.677767 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.678211 23633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.680624 23632 tablet_service.cc:1505] Processing CreateTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 (DEFAULT_TABLE table=default.Table_385b3d01f133412a978baad54da4880d [id=1d6f12be6ba342c98a64503cd58d8ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.681262 23632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0cbd6bec674cf5831486f500cc7b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.681380 23643 tablet_service.cc:1505] Processing CreateTablet for tablet e3df1fd83e3c4ac29a7846736701044a (DEFAULT_TABLE table=default.Table_3ad1b2da425540ccbe8b55d22f379b4a [id=7810e4d5193c4559891f1603fe361e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.681842 23643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3df1fd83e3c4ac29a7846736701044a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:22.680893 23605 leader_election.cc:343] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.682265 23605 leader_election.cc:304] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:22.682838 23148 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a570f339890946b7ab20c15de68462d5 consensus state has no leader
I20251023 20:15:22.683144 24174 raft_consensus.cc:2749] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:22.683211 23605 leader_election.cc:343] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.683211 23148 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_deb8b605c159407ab25c527a038c8d44" } modify_external_catalogs: true
I20251023 20:15:22.683310 23605 leader_election.cc:304] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:22.683408 23148 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40384:
table { table_name: "default.Table_deb8b605c159407ab25c527a038c8d44" } modify_external_catalogs: true
I20251023 20:15:22.681236 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 9749c6f3fe5545ab93bf49aae97e9818 (DEFAULT_TABLE table=default.Table_b589e93e3314462d8ef529985314e501 [id=bdeb476ba31d425893262ab98d5b579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.684581 23147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40384:
name: "default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:22.685170 23147 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:22.687287 24174 raft_consensus.cc:2749] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:22.687345 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 2e735945647148478201eeffa1d03632 (DEFAULT_TABLE table=default.Table_aba7529cde2c47e39145a97c70f643ed [id=c949af1c17974ec991f6bb58c01c4f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.687319 23636 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.687289 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 364e3b9a07e44c8992f6d9cbda6a86e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:22.689543 23142 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a744214a24cc4bf28654ffa417ce0d3d consensus state has no leader
I20251023 20:15:22.690686 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 (DEFAULT_TABLE table=default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:22.693254 23139 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3a216aeebe649f79b6d873e1c59df37 consensus state has no leader
I20251023 20:15:22.693280 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 660d84eb63a4490886aba1777d449ee2 (DEFAULT_TABLE table=default.Table_ec09021e59f8464f81cb6ceda1f1c567 [id=2fd818d424a44efdab6cf6fda7537ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.694224 23145 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet 8317a54a5f7b4afe8da9d336268efe43
I20251023 20:15:22.695327 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.695897 23635 tablet_service.cc:1505] Processing CreateTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 (DEFAULT_TABLE table=default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a [id=273447c687354c109e50ecd2bd95084b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.696167 23787 ts_tablet_manager.cc:1403] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.291s user 0.006s sys 0.001s
W20251023 20:15:22.699385 23605 consensus_peers.cc:597] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 -> Peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Couldn't send request to peer d220aba7c9c0400398ffde917376ee48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2273d5581784756b737f8c7c17ed3a9. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:22.704352 23790 tablet_bootstrap.cc:492] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.704530 23790 ts_tablet_manager.cc:1403] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.075s user 0.011s sys 0.007s
I20251023 20:15:22.705379 23790 raft_consensus.cc:359] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.705570 23790 raft_consensus.cc:385] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.705652 23790 raft_consensus.cc:740] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.705855 23790 consensus_queue.cc:260] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.707732 23790 ts_tablet_manager.cc:1434] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:22.708122 23790 tablet_bootstrap.cc:492] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.709050 23638 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.700295 23631 tablet_service.cc:1505] Processing CreateTablet for tablet 118be7aa6380469db091d2788472e62b (DEFAULT_TABLE table=default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.709929 23790 tablet_bootstrap.cc:654] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.711124 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.711870 23634 tablet_service.cc:1505] Processing CreateTablet for tablet 63ef1bae87c54aa988dfe49f828fe4f8 (DEFAULT_TABLE table=default.Table_6443f81020254f45a1b7fd93ad217e09 [id=4d43a0eda13041689a414ecbadb85e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.712255 23642 tablet_service.cc:1505] Processing CreateTablet for tablet a0079496170849c6b3c233e1cd78e40e (DEFAULT_TABLE table=default.Table_3a5e9b90e86340ed80fdb4a8679777e6 [id=22d88af65ebd43389f6e58573143816c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.712808 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.713340 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e735945647148478201eeffa1d03632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.714044 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f2a4f1d52984af58d6286671389b6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.718883 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9749c6f3fe5545ab93bf49aae97e9818. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.723227 23786 raft_consensus.cc:359] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.724025 23783 ts_tablet_manager.cc:1929] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.724182 23783 log.cc:1199] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/9fc7e18054c440098dc2470143ea7fae
I20251023 20:15:22.724529 23783 ts_tablet_manager.cc:1950] T 9fc7e18054c440098dc2470143ea7fae P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:22.724820 23144 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet aa794b3bec4b4b3993c814ea7cd02ee2
I20251023 20:15:22.725903 23140 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet ea0cbd6bec674cf5831486f500cc7b36
I20251023 20:15:22.726540 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660d84eb63a4490886aba1777d449ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.727281 23148 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Sending DeleteTablet for 3 replicas of tablet d951134f6773410386bc57f7f90310c3
I20251023 20:15:22.728659 23786 raft_consensus.cc:385] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.728783 23786 raft_consensus.cc:740] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.730436 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.731511 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.732928 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38382
W20251023 20:15:22.734759 23470 leader_election.cc:343] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1bcdec7c16c943079e20bb486aecb431
I20251023 20:15:22.735682 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.737763 23784 ts_tablet_manager.cc:1950] T 451955aee3aa4bcb82252bcefbbcaefe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:22.738173 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.738363 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.739032 23493 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 104) took 1521 ms. Trace:
W20251023 20:15:22.669073 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
W20251023 20:15:22.739284 23134 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 70238.6us
I20251023 20:15:22.739467 23786 consensus_queue.cc:260] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.740207 23786 ts_tablet_manager.cc:1434] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
I20251023 20:15:22.740312 23470 leader_election.cc:304] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:22.740612 24184 raft_consensus.cc:2749] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:22.741312 23494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 102) took 1633 ms. Trace:
I20251023 20:15:22.742772 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet e31aebfc87734ffeb2c0b25c07cb3f70 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
I20251023 20:15:22.743039 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 585a9af140d9479fb83d13e68a62300c (table default.Table_c3d2d37dd2bd414fae4c968fc3f52a25 [id=97396cb497da42f6af247dd68249182e]) successfully deleted
I20251023 20:15:22.744493 23787 raft_consensus.cc:359] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.744799 23492 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 103) took 1636 ms. Trace:
I20251023 20:15:22.744839 23787 raft_consensus.cc:385] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.745478 23787 raft_consensus.cc:740] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.746927 23635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8317a54a5f7b4afe8da9d336268efe43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.747699 23631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118be7aa6380469db091d2788472e62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.747805 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 451955aee3aa4bcb82252bcefbbcaefe (table default.Table_537d89cd0e14498085731d83eae39d0a [id=5533ec0faf2a4bf9abf3aa2c75d9b0ca]) successfully deleted
I20251023 20:15:22.748839 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 434) took 1875 ms. Trace:
I20251023 20:15:22.748945 23783 rpcz_store.cc:276] 1023 20:15:20.870839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.871607 (+ 768us) service_pool.cc:225] Handling call
1023 20:15:22.448219 (+1576612us) ts_tablet_manager.cc:1151] Deleting tablet 9fc7e18054c440098dc2470143ea7fae
1023 20:15:22.448230 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.677059 (+228829us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.723911 (+ 46852us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.745847 (+ 21936us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":223092,"tablet-delete.queue_time_us":1576475}
I20251023 20:15:22.749495 23133 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 9fc7e18054c440098dc2470143ea7fae (table default.Table_265f006bfcc2450f964410c799f035f9 [id=f0872a169a604701b06c246f1af67ac0]) successfully deleted
I20251023 20:15:22.743216 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d0e61de7a93d450d9ed79eef82d7e5e3 (table default.Table_4d358925f1254129a8b42bfca11f8870 [id=a1701c81a217439197ef53ff39675ef9]) successfully deleted
I20251023 20:15:22.749922 23784 tablet_replica.cc:333] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:22.750064 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a86a25659ec44d7c825783036342ed45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a86a25659ec44d7c825783036342ed45 already in progress: creating tablet
W20251023 20:15:22.750464 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 79e3d422a66f432789c0cf4a00e57af2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e3d422a66f432789c0cf4a00e57af2 already in progress: creating tablet
I20251023 20:15:22.750705 23495 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 101) took 1688 ms. Trace:
W20251023 20:15:22.750862 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 2fa6a838b1bd409ba2a4e2074576ee23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fa6a838b1bd409ba2a4e2074576ee23 already in progress: creating tablet
W20251023 20:15:22.751691 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
W20251023 20:15:22.752472 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c97e2e0f839a4ef389fe100b6a81301e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97e2e0f839a4ef389fe100b6a81301e already in progress: creating tablet
I20251023 20:15:22.739292 23493 rpcz_store.cc:276] 1023 20:15:21.217422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.217490 (+ 68us) service_pool.cc:225] Handling call
1023 20:15:21.712890 (+495400us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x6160001c2754. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:21.984618 (+271728us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x6160001c3354. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.063744 (+ 79126us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x616000171e30. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a5459229 00007fc3a7f80ff8 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.651537 (+587793us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x6160001b4c54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.729246 (+ 77709us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x6160001a7e30. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a5459229 00007fc3a7f80ff8 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.738709 (+ 9463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2240668416}
W20251023 20:15:22.753319 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet bb475e303d07491bbc19dc3f628bceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb475e303d07491bbc19dc3f628bceb7 already in progress: creating tablet
W20251023 20:15:22.754119 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 40e0301fb5974ec5b6e8291230a736c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40e0301fb5974ec5b6e8291230a736c2 already in progress: creating tablet
I20251023 20:15:22.754873 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 8e19f64a9ac44efba92928d0fa16819a (table default.Table_e58c00cb57f34786864684583ab1adf6 [id=aea6c365745a4e50a1768a10d7803d80]) successfully deleted
W20251023 20:15:22.756562 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 0858358aabff47a69dc77f4669b225c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0858358aabff47a69dc77f4669b225c5 already in progress: creating tablet
I20251023 20:15:22.757377 23494 rpcz_store.cc:276] 1023 20:15:21.108250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.108462 (+ 212us) service_pool.cc:225] Handling call
1023 20:15:21.712890 (+604428us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x6160001c2754. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:21.984482 (+271592us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x6160001c3354. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.653258 (+668776us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x6160001b4c54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.741278 (+ 88020us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1318033536}
I20251023 20:15:22.757802 23786 tablet_bootstrap.cc:492] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.757730 23492 rpcz_store.cc:276] 1023 20:15:21.108411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.108591 (+ 180us) service_pool.cc:225] Handling call
1023 20:15:21.712722 (+604131us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x6160001c2754. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:21.984789 (+272067us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x6160001c3354. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.651313 (+666524us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x6160001b4c54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.657304 (+ 5991us) spinlock_profiling.cc:243] Waited 928 us on lock 0x6160000d3d30. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a5459229 00007fc3a7f80ff8 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.744755 (+ 87451us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2031974528}
I20251023 20:15:22.758930 23790 tablet_bootstrap.cc:492] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.759120 23790 ts_tablet_manager.cc:1403] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.002s
I20251023 20:15:22.759932 23790 raft_consensus.cc:359] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.760087 23786 tablet_bootstrap.cc:654] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.760143 23790 raft_consensus.cc:385] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.760221 23790 raft_consensus.cc:740] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.760478 23790 consensus_queue.cc:260] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.761582 23783 tablet_replica.cc:333] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:22.761830 23783 raft_consensus.cc:2243] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.762267 23783 raft_consensus.cc:2272] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.762862 23783 ts_tablet_manager.cc:1916] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.762889 23790 ts_tablet_manager.cc:1434] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251023 20:15:22.764933 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ef1bae87c54aa988dfe49f828fe4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.763020 23787 consensus_queue.cc:260] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.757953 23784 raft_consensus.cc:2243] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.758019 23495 rpcz_store.cc:276] 1023 20:15:21.061869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.061925 (+ 56us) service_pool.cc:225] Handling call
1023 20:15:21.713166 (+651241us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x6160001c2754. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:21.978752 (+265586us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x6160001c3354. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.107864 (+129112us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x61800006088c. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a565de3e 00007fc3a54594c1 00007fc3a7f80ff8 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788
1023 20:15:22.650398 (+542534us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x6160001b4c54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:22.750660 (+100262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2014407680}
I20251023 20:15:22.766088 23784 raft_consensus.cc:2272] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.766635 23784 ts_tablet_manager.cc:1916] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.769625 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 6558265ff9e249c68123577958f1401a (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]) successfully deleted
W20251023 20:15:22.769872 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e418a62d00824cff89b37a7bace5c469 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e418a62d00824cff89b37a7bace5c469 already in progress: creating tablet
W20251023 20:15:22.770085 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d82c92e863b2470699cc822b8cb8c14d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d82c92e863b2470699cc822b8cb8c14d already in progress: creating tablet
W20251023 20:15:22.770267 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
W20251023 20:15:22.770457 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
W20251023 20:15:22.770641 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
I20251023 20:15:22.770807 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a2273d5581784756b737f8c7c17ed3a9 (table default.Table_156963dcf5454475a0cae4d52eff17f2 [id=9ed392ddcba74cd4a629eb0b728171ae]) successfully deleted
W20251023 20:15:22.770990 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf already in progress: creating tablet
W20251023 20:15:22.771175 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
I20251023 20:15:22.771421 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 3894dd4feeea4c16861a6c9520059474 (table default.Table_67a76ff63d4d411582ab8e10ac4f6924 [id=fb6dc9a69da44f35a0184490d90a1e5c]) successfully deleted
W20251023 20:15:22.771736 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
W20251023 20:15:22.771921 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
I20251023 20:15:22.772042 23038 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu pid 23114
W20251023 20:15:22.772121 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
W20251023 20:15:22.772290 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a8eaa2167f82475ca2f6a110aba288ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8eaa2167f82475ca2f6a110aba288ec already in progress: creating tablet
W20251023 20:15:22.772456 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
I20251023 20:15:22.772857 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 23225d31c2e647398aa58494c627507e (table default.Table_67a7910474894d659750844c77625a9d [id=82c65a189d13499c8471cb1ce0894c45]) successfully deleted
I20251023 20:15:22.777221 23790 tablet_bootstrap.cc:492] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.781117 23790 tablet_bootstrap.cc:654] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.781726 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38382
I20251023 20:15:22.788046 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0079496170849c6b3c233e1cd78e40e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:22.773064 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
I20251023 20:15:22.788337 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet e31aebfc87734ffeb2c0b25c07cb3f70 with delete_type TABLET_DATA_DELETED (replaced by tablet edec01a63a3046d9ae5a372fc9068b52) from {username='slave'} at 127.0.0.1:44712
W20251023 20:15:22.789284 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
W20251023 20:15:22.789566 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
I20251023 20:15:22.791559 23787 ts_tablet_manager.cc:1434] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.092s user 0.002s sys 0.000s
W20251023 20:15:22.791870 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
I20251023 20:15:22.793493 23784 ts_tablet_manager.cc:1929] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.793655 23784 log.cc:1199] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/1bcdec7c16c943079e20bb486aecb431
W20251023 20:15:22.793958 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 419e41d7a37f4b888cb242b089c28790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 419e41d7a37f4b888cb242b089c28790 already in progress: creating tablet
I20251023 20:15:22.794039 23784 ts_tablet_manager.cc:1950] T 1bcdec7c16c943079e20bb486aecb431 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:22.794387 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet e0f8d8184a68453abd2792ccc7740aa1 (table default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]) successfully deleted
W20251023 20:15:22.794685 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
W20251023 20:15:22.794948 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d9f422575c7c4251879130501ab721fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f422575c7c4251879130501ab721fd already in progress: creating tablet
W20251023 20:15:22.795518 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a9c5b69df4c04365b1ac1457fe8300cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9c5b69df4c04365b1ac1457fe8300cc already in progress: creating tablet
W20251023 20:15:22.796066 23134 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
I20251023 20:15:22.796974 23790 tablet_bootstrap.cc:492] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.797255 23790 ts_tablet_manager.cc:1403] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20251023 20:15:22.798209 23790 raft_consensus.cc:359] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.798455 23790 raft_consensus.cc:385] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.798617 23790 raft_consensus.cc:740] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.798916 23790 consensus_queue.cc:260] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.799376 23787 tablet_bootstrap.cc:492] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:22.800971 23790 ts_tablet_manager.cc:1434] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:22.797900 23133 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 1bcdec7c16c943079e20bb486aecb431 (table default.Table_6f178deaeeb94c4da6a40568c635e0b5 [id=cc451577eb8b4ba7937b8dbe4e859bc8]) successfully deleted
W20251023 20:15:22.804744 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
W20251023 20:15:22.805348 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
W20251023 20:15:22.806233 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
W20251023 20:15:22.807888 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 419e41d7a37f4b888cb242b089c28790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 419e41d7a37f4b888cb242b089c28790 already in progress: creating tablet
W20251023 20:15:22.808640 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
I20251023 20:15:22.809130 23790 tablet_bootstrap.cc:492] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.809283 23134 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 364e3b9a07e44c8992f6d9cbda6a86e5 (table default.Table_677942097b3a441287c448bb7c584a4f [id=7fa58e925ef94309ba20deab24c9be99]) successfully deleted
I20251023 20:15:22.811224 23790 tablet_bootstrap.cc:654] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:22.809633 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
W20251023 20:15:22.812024 23133 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9f422575c7c4251879130501ab721fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f422575c7c4251879130501ab721fd already in progress: creating tablet
I20251023 20:15:22.817488 23784 tablet_replica.cc:333] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:22.818063 23784 raft_consensus.cc:2243] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.818840 23784 raft_consensus.cc:2272] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.820752 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9b192c111c47d1983e9a48d07da288 (DEFAULT_TABLE table=default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d [id=5a207d264756433f94af5164462692ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.821828 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9b192c111c47d1983e9a48d07da288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.821772 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 1e19859a65a144598e1612be89daa3a1 (DEFAULT_TABLE table=default.Table_1ec08f875c4f4a438cd2081db4326eb3 [id=0aefe460f19449a08ef7574681b2af94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.822256 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e19859a65a144598e1612be89daa3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.812407 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8 (DEFAULT_TABLE table=default.Table_10bf526d2893408887984427efbaa1f2 [id=933b756e69bf4d6d96590acd69f45267]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.829312 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.829569 23784 ts_tablet_manager.cc:1916] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.830265 23786 tablet_bootstrap.cc:492] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.830495 23786 ts_tablet_manager.cc:1403] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.005s
I20251023 20:15:22.831261 23786 raft_consensus.cc:359] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.831523 23786 raft_consensus.cc:385] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.831652 23786 raft_consensus.cc:740] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.831923 23786 consensus_queue.cc:260] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.812808 23374 tablet_service.cc:1505] Processing CreateTablet for tablet a0079496170849c6b3c233e1cd78e40e (DEFAULT_TABLE table=default.Table_3a5e9b90e86340ed80fdb4a8679777e6 [id=22d88af65ebd43389f6e58573143816c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.833429 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0079496170849c6b3c233e1cd78e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.834352 23786 ts_tablet_manager.cc:1434] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:22.834626 23786 tablet_bootstrap.cc:492] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.836347 23786 tablet_bootstrap.cc:654] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.817854 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 660d84eb63a4490886aba1777d449ee2 (DEFAULT_TABLE table=default.Table_ec09021e59f8464f81cb6ceda1f1c567 [id=2fd818d424a44efdab6cf6fda7537ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.837386 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660d84eb63a4490886aba1777d449ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.818580 23373 tablet_service.cc:1505] Processing CreateTablet for tablet 2e735945647148478201eeffa1d03632 (DEFAULT_TABLE table=default.Table_aba7529cde2c47e39145a97c70f643ed [id=c949af1c17974ec991f6bb58c01c4f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.820624 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 255521d943d34713aaf773b15f9168c6 (DEFAULT_TABLE table=default.Table_1f76833d42104cee91c1406f649fa028 [id=a48bcc6da6504ccb941c18dd5e289c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.821013 23366 tablet_service.cc:1505] Processing CreateTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 (DEFAULT_TABLE table=default.Table_385b3d01f133412a978baad54da4880d [id=1d6f12be6ba342c98a64503cd58d8ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.841674 23373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e735945647148478201eeffa1d03632. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.841914 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0cbd6bec674cf5831486f500cc7b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.843534 23631 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9b192c111c47d1983e9a48d07da288 (DEFAULT_TABLE table=default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d [id=5a207d264756433f94af5164462692ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.843938 23631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9b192c111c47d1983e9a48d07da288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.822012 23369 tablet_service.cc:1505] Processing CreateTablet for tablet e3df1fd83e3c4ac29a7846736701044a (DEFAULT_TABLE table=default.Table_3ad1b2da425540ccbe8b55d22f379b4a [id=7810e4d5193c4559891f1603fe361e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.845419 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3df1fd83e3c4ac29a7846736701044a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.822620 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 9749c6f3fe5545ab93bf49aae97e9818 (DEFAULT_TABLE table=default.Table_b589e93e3314462d8ef529985314e501 [id=bdeb476ba31d425893262ab98d5b579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.823647 23372 tablet_service.cc:1505] Processing CreateTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 (DEFAULT_TABLE table=default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.849460 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9749c6f3fe5545ab93bf49aae97e9818. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.849713 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f2a4f1d52984af58d6286671389b6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.850577 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255521d943d34713aaf773b15f9168c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.850596 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 1e19859a65a144598e1612be89daa3a1 (DEFAULT_TABLE table=default.Table_1ec08f875c4f4a438cd2081db4326eb3 [id=0aefe460f19449a08ef7574681b2af94]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.851115 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e19859a65a144598e1612be89daa3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.853840 23784 ts_tablet_manager.cc:1929] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:22.860874 23784 log.cc:1199] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/364e3b9a07e44c8992f6d9cbda6a86e5
I20251023 20:15:22.861502 23784 ts_tablet_manager.cc:1950] T 364e3b9a07e44c8992f6d9cbda6a86e5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:22.862432 23786 tablet_bootstrap.cc:492] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.862720 23786 ts_tablet_manager.cc:1403] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20251023 20:15:22.863674 23786 raft_consensus.cc:359] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.863969 23786 raft_consensus.cc:385] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.864084 23786 raft_consensus.cc:740] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.864382 23786 consensus_queue.cc:260] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.870012 23786 ts_tablet_manager.cc:1434] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:22.872468 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 5d801ca6ce63440abefe0c694889fa9e (DEFAULT_TABLE table=default.Table_4fb0e33f75594ca59d2ecb391873d804 [id=1a32884612a34df48b257e1a18bb5e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.872489 23650 tablet_service.cc:1505] Processing CreateTablet for tablet f4023a5776334f42a2564f538a7c5fee (DEFAULT_TABLE table=default.Table_527cb9b18f414f43872b4f9dd4c433e1 [id=cf913261a84244e0a0ef54347a3a548c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.872881 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4023a5776334f42a2564f538a7c5fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.872907 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d801ca6ce63440abefe0c694889fa9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.873380 23649 tablet_service.cc:1505] Processing CreateTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 (DEFAULT_TABLE table=default.Table_20216f4bd8cd4150bfb55aa1e918e7c7 [id=ad2f114d6aaa4bc9a4305c650c90bb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.873715 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5acdc66df9c748a1a6300962f62fcf36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.874286 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.875710 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.876454 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.877453 23502 tablet_service.cc:1505] Processing CreateTablet for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (DEFAULT_TABLE table=default.Table_e6177be8581a457cb9113fcc941d6c65 [id=edda395d9ca84731a2bdc438a41aca1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.877620 23634 tablet_service.cc:1505] Processing CreateTablet for tablet 5d801ca6ce63440abefe0c694889fa9e (DEFAULT_TABLE table=default.Table_4fb0e33f75594ca59d2ecb391873d804 [id=1a32884612a34df48b257e1a18bb5e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.877830 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a74b3f22b854c1ab2b6cc817b84cf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.877954 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d801ca6ce63440abefe0c694889fa9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.881388 23635 tablet_service.cc:1505] Processing CreateTablet for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (DEFAULT_TABLE table=default.Table_e6177be8581a457cb9113fcc941d6c65 [id=edda395d9ca84731a2bdc438a41aca1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.881794 23635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a74b3f22b854c1ab2b6cc817b84cf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.884639 23514 tablet_service.cc:1505] Processing CreateTablet for tablet f4023a5776334f42a2564f538a7c5fee (DEFAULT_TABLE table=default.Table_527cb9b18f414f43872b4f9dd4c433e1 [id=cf913261a84244e0a0ef54347a3a548c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.885022 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4023a5776334f42a2564f538a7c5fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.885676 23178 generic_service.cc:196] Checking for leaks (request via RPC)
I20251023 20:15:22.886318 23787 tablet_bootstrap.cc:654] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.888373 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 (DEFAULT_TABLE table=default.Table_20216f4bd8cd4150bfb55aa1e918e7c7 [id=ad2f114d6aaa4bc9a4305c650c90bb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:22.888818 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5acdc66df9c748a1a6300962f62fcf36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:22.900105 23786 tablet_bootstrap.cc:492] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.902282 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:47314
I20251023 20:15:22.902503 24186 raft_consensus.cc:493] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:22.902675 24186 raft_consensus.cc:515] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.903321 24186 leader_election.cc:290] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:22.904148 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:22.904371 23670 raft_consensus.cc:2468] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:22.904579 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:22.904964 23333 leader_election.cc:304] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:22.905428 24186 raft_consensus.cc:2804] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:22.905529 24186 raft_consensus.cc:493] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:22.905635 24186 raft_consensus.cc:3060] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:22.905615 23333 leader_election.cc:343] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.906962 23786 tablet_bootstrap.cc:654] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.907994 23783 ts_tablet_manager.cc:1929] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.908109 23783 log.cc:1199] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/ef1229b9e0384a82811f244d2bf4c8ce
I20251023 20:15:22.908233 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e19f64a9ac44efba92928d0fa16819a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:44712
I20251023 20:15:22.908474 23783 ts_tablet_manager.cc:1950] T ef1229b9e0384a82811f244d2bf4c8ce P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:22.908452 24186 raft_consensus.cc:515] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.909152 24186 leader_election.cc:290] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:22.909204 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 435) took 2038 ms. Trace:
I20251023 20:15:22.909322 23783 rpcz_store.cc:276] 1023 20:15:20.870959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.871766 (+ 807us) service_pool.cc:225] Handling call
1023 20:15:22.749257 (+1877491us) ts_tablet_manager.cc:1151] Deleting tablet ef1229b9e0384a82811f244d2bf4c8ce
1023 20:15:22.749267 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.865410 (+116143us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.904022 (+ 38612us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.909164 (+ 5142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1877341}
I20251023 20:15:22.909624 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:22.909723 23784 tablet_replica.cc:333] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:22.909809 23670 raft_consensus.cc:3060] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:22.909853 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:22.910321 23333 leader_election.cc:343] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.909920 23784 raft_consensus.cc:2243] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.910574 23784 raft_consensus.cc:2272] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.911281 23784 ts_tablet_manager.cc:1916] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.912531 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 437) took 2036 ms. Trace:
I20251023 20:15:22.912652 23783 rpcz_store.cc:276] 1023 20:15:20.876130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.876196 (+ 66us) service_pool.cc:225] Handling call
1023 20:15:22.912360 (+2036164us) ts_tablet_manager.cc:1151] Deleting tablet 79e3d422a66f432789c0cf4a00e57af2
1023 20:15:22.912369 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.912482 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035930}
I20251023 20:15:22.913044 23670 raft_consensus.cc:2468] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:22.913563 23043 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 57036.3us
I20251023 20:15:22.913846 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 436) took 2037 ms. Trace:
I20251023 20:15:22.913911 23333 leader_election.cc:304] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:22.913937 23783 rpcz_store.cc:276] 1023 20:15:20.876008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.876420 (+ 412us) service_pool.cc:225] Handling call
1023 20:15:22.913687 (+2037267us) ts_tablet_manager.cc:1151] Deleting tablet a86a25659ec44d7c825783036342ed45
1023 20:15:22.913696 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.913817 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037109}
I20251023 20:15:22.914407 24186 raft_consensus.cc:2804] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:22.914963 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 438) took 2024 ms. Trace:
I20251023 20:15:22.915077 23783 rpcz_store.cc:276] 1023 20:15:20.890573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.890938 (+ 365us) service_pool.cc:225] Handling call
1023 20:15:22.914764 (+2023826us) ts_tablet_manager.cc:1151] Deleting tablet 2fa6a838b1bd409ba2a4e2074576ee23
1023 20:15:22.914772 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.914923 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68,"tablet-delete.queue_time_us":2023637}
I20251023 20:15:22.916090 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 450) took 1947 ms. Trace:
I20251023 20:15:22.916208 23783 rpcz_store.cc:276] 1023 20:15:20.969008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.969054 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:22.915953 (+1946899us) ts_tablet_manager.cc:1151] Deleting tablet ca11ddd5d97544cf916b87fd3dff219d
1023 20:15:22.915963 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.916041 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936464}
I20251023 20:15:22.917364 24186 raft_consensus.cc:697] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:22.917716 24186 consensus_queue.cc:237] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.918424 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 451) took 1949 ms. Trace:
I20251023 20:15:22.918558 23783 rpcz_store.cc:276] 1023 20:15:20.969337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.969397 (+ 60us) service_pool.cc:225] Handling call
1023 20:15:22.918279 (+1948882us) ts_tablet_manager.cc:1151] Deleting tablet c97e2e0f839a4ef389fe100b6a81301e
1023 20:15:22.918289 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.918389 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938687}
I20251023 20:15:22.919514 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 452) took 1950 ms. Trace:
I20251023 20:15:22.919641 23783 rpcz_store.cc:276] 1023 20:15:20.969438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:20.970110 (+ 672us) service_pool.cc:225] Handling call
1023 20:15:22.919392 (+1949282us) ts_tablet_manager.cc:1151] Deleting tablet bb475e303d07491bbc19dc3f628bceb7
1023 20:15:22.919401 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.919478 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1939760}
I20251023 20:15:22.920711 23786 tablet_bootstrap.cc:492] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.920936 23786 ts_tablet_manager.cc:1403] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20251023 20:15:22.921658 23790 tablet_bootstrap.cc:492] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.921828 23786 raft_consensus.cc:359] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.922013 23786 raft_consensus.cc:385] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.922104 23786 raft_consensus.cc:740] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.922395 23786 consensus_queue.cc:260] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.922708 23790 ts_tablet_manager.cc:1403] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.121s user 0.000s sys 0.007s
I20251023 20:15:22.922981 23783 tablet_replica.cc:333] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:22.923183 23783 raft_consensus.cc:2243] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.923331 23783 raft_consensus.cc:2272] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.923823 23783 ts_tablet_manager.cc:1916] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.924460 23790 raft_consensus.cc:359] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.925693 23790 raft_consensus.cc:385] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.925963 23790 raft_consensus.cc:740] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.926446 23786 ts_tablet_manager.cc:1434] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251023 20:15:22.926441 23790 consensus_queue.cc:260] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.929998 23790 ts_tablet_manager.cc:1434] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251023 20:15:22.931689 23790 tablet_bootstrap.cc:492] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.935606 23787 tablet_bootstrap.cc:492] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:22.935830 23787 ts_tablet_manager.cc:1403] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.143s user 0.009s sys 0.008s
I20251023 20:15:22.936640 23787 raft_consensus.cc:359] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.936900 23787 raft_consensus.cc:385] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.937000 23787 raft_consensus.cc:740] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.937057 23786 tablet_bootstrap.cc:492] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.937408 23787 consensus_queue.cc:260] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.937654 23790 tablet_bootstrap.cc:654] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.939138 23786 tablet_bootstrap.cc:654] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.940462 23784 ts_tablet_manager.cc:1929] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.941226 23783 ts_tablet_manager.cc:1929] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.941354 23783 log.cc:1199] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/3fbad98d3ab24cc5b7726a92fba04cf8
I20251023 20:15:22.941671 23783 ts_tablet_manager.cc:1950] T 3fbad98d3ab24cc5b7726a92fba04cf8 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:22.942338 23784 log.cc:1199] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e31aebfc87734ffeb2c0b25c07cb3f70
I20251023 20:15:22.942797 23784 ts_tablet_manager.cc:1950] T e31aebfc87734ffeb2c0b25c07cb3f70 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:22.942991 23787 ts_tablet_manager.cc:1434] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:22.944778 23784 tablet_replica.cc:333] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:22.944947 23784 raft_consensus.cc:2243] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.945209 23784 raft_consensus.cc:2272] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.946237 23786 tablet_bootstrap.cc:492] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.946393 23786 ts_tablet_manager.cc:1403] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251023 20:15:22.945216 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 457) took 1942 ms. Trace:
I20251023 20:15:22.946138 23787 tablet_bootstrap.cc:492] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:22.946943 23783 rpcz_store.cc:276] 1023 20:15:21.003017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.003071 (+ 54us) service_pool.cc:225] Handling call
1023 20:15:22.922917 (+1919846us) ts_tablet_manager.cc:1151] Deleting tablet 3fbad98d3ab24cc5b7726a92fba04cf8
1023 20:15:22.922926 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.926929 (+ 4003us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.941121 (+ 14192us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.945165 (+ 4044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919623}
I20251023 20:15:22.947366 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 458) took 1930 ms. Trace:
I20251023 20:15:22.947453 23783 rpcz_store.cc:276] 1023 20:15:21.016841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.016891 (+ 50us) service_pool.cc:225] Handling call
1023 20:15:22.947200 (+1930309us) ts_tablet_manager.cc:1151] Deleting tablet 40e0301fb5974ec5b6e8291230a736c2
1023 20:15:22.947209 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.947312 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927160}
I20251023 20:15:22.947788 23783 tablet_replica.cc:333] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:22.947964 23783 raft_consensus.cc:2243] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:22.948143 23783 raft_consensus.cc:2272] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:22.948675 23786 raft_consensus.cc:359] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.948864 23786 raft_consensus.cc:385] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.948923 23786 raft_consensus.cc:740] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.949190 23786 consensus_queue.cc:260] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.951364 23786 ts_tablet_manager.cc:1434] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:22.953411 23786 tablet_bootstrap.cc:492] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.955080 23786 tablet_bootstrap.cc:654] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.956374 23784 ts_tablet_manager.cc:1916] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.956755 23787 tablet_bootstrap.cc:654] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.957043 23783 ts_tablet_manager.cc:1916] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:22.957984 23790 tablet_bootstrap.cc:492] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.958191 23790 ts_tablet_manager.cc:1403] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20251023 20:15:22.959044 23790 raft_consensus.cc:359] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.959301 23790 raft_consensus.cc:385] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.959430 23790 raft_consensus.cc:740] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.959713 23790 consensus_queue.cc:260] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.963696 23784 ts_tablet_manager.cc:1929] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.965134 23784 log.cc:1199] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/8e19f64a9ac44efba92928d0fa16819a
I20251023 20:15:22.965193 23790 ts_tablet_manager.cc:1434] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:22.965773 23784 ts_tablet_manager.cc:1950] T 8e19f64a9ac44efba92928d0fa16819a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:22.967948 23786 tablet_bootstrap.cc:492] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.968160 23786 ts_tablet_manager.cc:1403] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20251023 20:15:22.969009 23786 raft_consensus.cc:359] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.969305 23786 raft_consensus.cc:385] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.969458 23786 raft_consensus.cc:740] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.969702 23790 tablet_bootstrap.cc:492] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:22.969838 23786 consensus_queue.cc:260] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:22.971654 23790 tablet_bootstrap.cc:654] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.973023 23786 ts_tablet_manager.cc:1434] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:22.975447 23786 tablet_bootstrap.cc:492] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.976087 23783 ts_tablet_manager.cc:1929] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:22.976226 23783 log.cc:1199] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/f1a6fe726ef844829a7ffec9d4494637
I20251023 20:15:22.976583 23783 ts_tablet_manager.cc:1950] T f1a6fe726ef844829a7ffec9d4494637 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:22.977111 23786 tablet_bootstrap.cc:654] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.977402 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 459) took 1881 ms. Trace:
I20251023 20:15:22.977527 23783 rpcz_store.cc:276] 1023 20:15:21.096273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.096370 (+ 97us) service_pool.cc:225] Handling call
1023 20:15:22.947699 (+1851329us) ts_tablet_manager.cc:1151] Deleting tablet f1a6fe726ef844829a7ffec9d4494637
1023 20:15:22.947707 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.963974 (+ 16267us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.975932 (+ 11958us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:22.977366 (+ 1434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1850783}
I20251023 20:15:22.979279 23787 tablet_bootstrap.cc:492] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:22.979485 23787 ts_tablet_manager.cc:1403] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.007s
I20251023 20:15:22.980315 23787 raft_consensus.cc:359] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.980540 23787 raft_consensus.cc:385] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.980664 23787 raft_consensus.cc:740] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.980968 23787 consensus_queue.cc:260] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.982437 23786 tablet_bootstrap.cc:492] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:22.982609 23786 ts_tablet_manager.cc:1403] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20251023 20:15:22.983342 23786 raft_consensus.cc:359] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.983522 23786 raft_consensus.cc:385] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.983587 23786 raft_consensus.cc:740] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.985427 23786 consensus_queue.cc:260] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.988448 23786 ts_tablet_manager.cc:1434] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251023 20:15:22.988445 23787 ts_tablet_manager.cc:1434] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:22.989482 23790 tablet_bootstrap.cc:492] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:22.989753 23790 ts_tablet_manager.cc:1403] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20251023 20:15:22.990679 23790 raft_consensus.cc:359] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.990978 23786 tablet_bootstrap.cc:492] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:22.990973 23790 raft_consensus.cc:385] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:22.991148 23790 raft_consensus.cc:740] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:22.991457 23790 consensus_queue.cc:260] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.992677 23786 tablet_bootstrap.cc:654] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:22.995450 24220 raft_consensus.cc:493] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:22.995599 24220 raft_consensus.cc:515] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:22.995767 23787 tablet_bootstrap.cc:492] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:22.996381 24220 leader_election.cc:290] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:22.997684 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:22.997990 23670 raft_consensus.cc:2468] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:22.998231 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 474) took 1858 ms. Trace:
I20251023 20:15:22.998343 23783 rpcz_store.cc:276] 1023 20:15:21.139942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.283693 (+143751us) service_pool.cc:225] Handling call
1023 20:15:22.978336 (+1694643us) ts_tablet_manager.cc:1151] Deleting tablet 0858358aabff47a69dc77f4669b225c5
1023 20:15:22.978344 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.998185 (+ 19841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612891}
I20251023 20:15:22.998396 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:22.998454 23333 leader_election.cc:304] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:22.998999 23787 tablet_bootstrap.cc:654] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:22.998965 23333 leader_election.cc:343] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:22.999584 24220 raft_consensus.cc:2804] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:22.999701 24220 raft_consensus.cc:493] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:22.999779 24220 raft_consensus.cc:3060] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:22.999981 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 478) took 1620 ms. Trace:
I20251023 20:15:23.000082 23783 rpcz_store.cc:276] 1023 20:15:21.391247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.391303 (+ 56us) service_pool.cc:225] Handling call
1023 20:15:22.999838 (+1608535us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:22.999848 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:22.999952 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540905}
I20251023 20:15:23.000521 23783 tablet_replica.cc:333] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:23.000717 23783 raft_consensus.cc:2243] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:23.000880 23783 raft_consensus.cc:2272] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:23.001706 23783 ts_tablet_manager.cc:1916] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:23.003079 24220 raft_consensus.cc:515] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.003618 23786 tablet_bootstrap.cc:492] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.003774 23786 ts_tablet_manager.cc:1403] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20251023 20:15:23.004362 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.004575 23670 raft_consensus.cc:3060] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.004544 23786 raft_consensus.cc:359] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.004712 23786 raft_consensus.cc:385] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.004802 23786 raft_consensus.cc:740] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.005034 23786 consensus_queue.cc:260] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.006724 23786 ts_tablet_manager.cc:1434] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:23.007021 23786 tablet_bootstrap.cc:492] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.007380 23670 raft_consensus.cc:2468] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.008427 23333 leader_election.cc:304] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.009331 24186 raft_consensus.cc:2804] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.011693 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:23.012261 23333 leader_election.cc:343] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.012513 23786 tablet_bootstrap.cc:654] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.012737 24220 leader_election.cc:290] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.015677 23783 ts_tablet_manager.cc:1929] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:23.015826 23783 log.cc:1199] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e418a62d00824cff89b37a7bace5c469
I20251023 20:15:23.016144 23783 ts_tablet_manager.cc:1950] T e418a62d00824cff89b37a7bace5c469 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:23.016861 23790 ts_tablet_manager.cc:1434] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20251023 20:15:23.016950 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 481) took 1603 ms. Trace:
I20251023 20:15:23.017068 23783 rpcz_store.cc:276] 1023 20:15:21.413160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.469772 (+ 56612us) service_pool.cc:225] Handling call
1023 20:15:23.000445 (+1530673us) ts_tablet_manager.cc:1151] Deleting tablet e418a62d00824cff89b37a7bace5c469
1023 20:15:23.000454 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.005039 (+ 4585us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.015543 (+ 10504us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.016888 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1530466}
I20251023 20:15:23.017868 23790 tablet_bootstrap.cc:492] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.018085 23787 tablet_bootstrap.cc:492] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.018299 23787 ts_tablet_manager.cc:1403] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.002s
I20251023 20:15:23.019169 23787 raft_consensus.cc:359] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.019544 23787 raft_consensus.cc:385] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.019659 23787 raft_consensus.cc:740] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.019981 23787 consensus_queue.cc:260] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.020181 23790 tablet_bootstrap.cc:654] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.024972 23787 ts_tablet_manager.cc:1434] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:23.027706 23786 tablet_bootstrap.cc:492] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.027927 23786 ts_tablet_manager.cc:1403] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.004s
I20251023 20:15:23.028638 23787 tablet_bootstrap.cc:492] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.029254 23786 raft_consensus.cc:359] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.029492 23786 raft_consensus.cc:385] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.029593 23786 raft_consensus.cc:740] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.029857 23786 consensus_queue.cc:260] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.029389 24186 raft_consensus.cc:697] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.030885 24186 consensus_queue.cc:237] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.031483 23786 ts_tablet_manager.cc:1434] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:23.035632 23786 tablet_bootstrap.cc:492] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.036211 23787 tablet_bootstrap.cc:654] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.037950 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 483) took 1623 ms. Trace:
I20251023 20:15:23.038134 23783 rpcz_store.cc:276] 1023 20:15:21.414092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.470606 (+ 56514us) service_pool.cc:225] Handling call
1023 20:15:23.037760 (+1567154us) ts_tablet_manager.cc:1151] Deleting tablet d82c92e863b2470699cc822b8cb8c14d
1023 20:15:23.037770 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.037897 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1552711}
I20251023 20:15:23.038851 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 489) took 1499 ms. Trace:
I20251023 20:15:23.038875 23786 tablet_bootstrap.cc:654] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.039012 23783 rpcz_store.cc:276] 1023 20:15:21.539845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.575595 (+ 35750us) service_pool.cc:225] Handling call
1023 20:15:23.038664 (+1463069us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:23.038674 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.038808 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451581}
I20251023 20:15:23.039546 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 491) took 1489 ms. Trace:
I20251023 20:15:23.039673 23783 rpcz_store.cc:276] 1023 20:15:21.550247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.615025 (+ 64778us) service_pool.cc:225] Handling call
1023 20:15:23.039406 (+1424381us) ts_tablet_manager.cc:1151] Deleting tablet 9ed0aa7e36e64843ad6b2ad4b415832a
1023 20:15:23.039415 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.039502 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371239}
I20251023 20:15:23.040115 23783 tablet_replica.cc:333] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:23.040354 23783 raft_consensus.cc:2243] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:23.040868 23783 raft_consensus.cc:2272] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:23.041867 23783 ts_tablet_manager.cc:1916] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:23.049381 24232 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae5a310b4a514c0f812bda133c756062)
I20251023 20:15:23.049690 24232 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:23.052438 24186 raft_consensus.cc:493] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.053404 24186 raft_consensus.cc:515] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:23.055160 23333 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:23.055994 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83da09ee0d10435ba89a19dc378d34eb" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:23.056653 23333 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.056793 23333 leader_election.cc:304] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.057175 24230 raft_consensus.cc:2749] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.057466 24186 leader_election.cc:290] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.058650 24232 leader_election.cc:290] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:23.058820 23786 tablet_bootstrap.cc:492] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.058952 23227 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 633 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a" is_pre_election: true
I20251023 20:15:23.059026 23786 ts_tablet_manager.cc:1403] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20251023 20:15:23.059857 23786 raft_consensus.cc:359] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.060014 23786 raft_consensus.cc:385] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.060104 23786 raft_consensus.cc:740] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.060338 23786 consensus_queue.cc:260] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.060813 23786 ts_tablet_manager.cc:1434] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:23.061582 23786 tablet_bootstrap.cc:492] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.063375 23786 tablet_bootstrap.cc:654] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.065914 23790 tablet_bootstrap.cc:492] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.066095 23790 ts_tablet_manager.cc:1403] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.005s
I20251023 20:15:23.066983 23790 raft_consensus.cc:359] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.067173 23790 raft_consensus.cc:385] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.067307 23790 raft_consensus.cc:740] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.067575 23790 consensus_queue.cc:260] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.068238 23790 ts_tablet_manager.cc:1434] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:23.069443 23790 tablet_bootstrap.cc:492] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.071259 23790 tablet_bootstrap.cc:654] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.078004 23786 tablet_bootstrap.cc:492] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.078205 23786 ts_tablet_manager.cc:1403] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251023 20:15:23.078979 23786 raft_consensus.cc:359] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.079175 23786 raft_consensus.cc:385] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.079248 23786 raft_consensus.cc:740] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.079465 23786 consensus_queue.cc:260] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.079931 23786 ts_tablet_manager.cc:1434] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:23.080485 23786 tablet_bootstrap.cc:492] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.082206 23786 tablet_bootstrap.cc:654] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.083845 23790 tablet_bootstrap.cc:492] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.083994 23787 tablet_bootstrap.cc:492] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.084084 23790 ts_tablet_manager.cc:1403] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20251023 20:15:23.084256 23787 ts_tablet_manager.cc:1403] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.000s
I20251023 20:15:23.084880 23790 raft_consensus.cc:359] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.085176 23790 raft_consensus.cc:385] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.085269 23790 raft_consensus.cc:740] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.085228 23787 raft_consensus.cc:359] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.085567 23787 raft_consensus.cc:385] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.085703 23787 raft_consensus.cc:740] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.086184 23790 consensus_queue.cc:260] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.086210 23787 consensus_queue.cc:260] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.089124 23787 ts_tablet_manager.cc:1434] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:23.089462 23787 tablet_bootstrap.cc:492] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.090009 23790 ts_tablet_manager.cc:1434] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251023 20:15:23.090335 23790 tablet_bootstrap.cc:492] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.096980 23790 tablet_bootstrap.cc:654] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.098928 23786 tablet_bootstrap.cc:492] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.099105 23786 ts_tablet_manager.cc:1403] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251023 20:15:23.100062 23786 raft_consensus.cc:359] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.100263 23786 raft_consensus.cc:385] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.101799 23787 tablet_bootstrap.cc:654] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.103789 23786 raft_consensus.cc:740] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.104076 23786 consensus_queue.cc:260] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.104640 23786 ts_tablet_manager.cc:1434] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:23.105791 23783 ts_tablet_manager.cc:1929] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:23.105914 23783 log.cc:1199] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a2273d5581784756b737f8c7c17ed3a9
I20251023 20:15:23.106335 23783 ts_tablet_manager.cc:1950] T a2273d5581784756b737f8c7c17ed3a9 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:23.107064 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 493) took 1538 ms. Trace:
I20251023 20:15:23.107193 23783 rpcz_store.cc:276] 1023 20:15:21.568614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.668199 (+ 99585us) service_pool.cc:225] Handling call
1023 20:15:23.040042 (+1371843us) ts_tablet_manager.cc:1151] Deleting tablet a2273d5581784756b737f8c7c17ed3a9
1023 20:15:23.040051 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.102508 (+ 62457us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.105653 (+ 3145us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.107025 (+ 1372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1353122}
I20251023 20:15:23.108453 23790 tablet_bootstrap.cc:492] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.108639 23790 ts_tablet_manager.cc:1403] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20251023 20:15:23.109534 23790 raft_consensus.cc:359] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.109731 23790 raft_consensus.cc:385] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.109822 23790 raft_consensus.cc:740] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.110050 23790 consensus_queue.cc:260] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.110237 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 500) took 1500 ms. Trace:
I20251023 20:15:23.111572 23786 tablet_bootstrap.cc:492] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.113380 23786 tablet_bootstrap.cc:654] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.113413 23783 rpcz_store.cc:276] 1023 20:15:21.609317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.843290 (+233973us) service_pool.cc:225] Handling call
1023 20:15:23.110075 (+1266785us) ts_tablet_manager.cc:1151] Deleting tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf
1023 20:15:23.110085 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.110199 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1243810}
I20251023 20:15:23.117288 23783 tablet_replica.cc:333] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:23.117586 23783 raft_consensus.cc:2243] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:23.117841 23783 raft_consensus.cc:2272] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:23.118474 23783 ts_tablet_manager.cc:1916] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:23.121121 23787 tablet_bootstrap.cc:492] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.121126 23790 ts_tablet_manager.cc:1434] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251023 20:15:23.121548 23790 tablet_bootstrap.cc:492] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.123406 23790 tablet_bootstrap.cc:654] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.124204 23787 ts_tablet_manager.cc:1403] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.002s
I20251023 20:15:23.125028 23787 raft_consensus.cc:359] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.125316 23787 raft_consensus.cc:385] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.125424 23787 raft_consensus.cc:740] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.125769 23787 consensus_queue.cc:260] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.128381 24198 raft_consensus.cc:493] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.128547 24198 raft_consensus.cc:515] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.129347 24198 leader_election.cc:290] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.129576 23787 ts_tablet_manager.cc:1434] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251023 20:15:23.130159 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.130985 23787 tablet_bootstrap.cc:492] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:23.131511 23604 leader_election.cc:343] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.130625 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.135293 23787 tablet_bootstrap.cc:654] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.138440 23790 tablet_bootstrap.cc:492] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:23.135604 23605 leader_election.cc:343] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.138736 23790 ts_tablet_manager.cc:1403] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20251023 20:15:23.138803 23605 leader_election.cc:304] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.139380 24198 raft_consensus.cc:2749] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.139595 23790 raft_consensus.cc:359] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.139894 23790 raft_consensus.cc:385] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.140014 23790 raft_consensus.cc:740] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.140334 23790 consensus_queue.cc:260] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.141150 23790 ts_tablet_manager.cc:1434] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.141841 24198 raft_consensus.cc:493] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.141975 24198 raft_consensus.cc:515] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.142622 24198 leader_election.cc:290] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.142781 23790 tablet_bootstrap.cc:492] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.144253 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "283166bbd8004ce3bca9f2ecbdbcbf48" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.144595 23790 tablet_bootstrap.cc:654] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.145222 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "283166bbd8004ce3bca9f2ecbdbcbf48" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:23.145695 23605 leader_election.cc:343] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:23.145989 23604 leader_election.cc:343] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.146097 23604 leader_election.cc:304] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.147903 24198 raft_consensus.cc:2749] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.154443 23783 ts_tablet_manager.cc:1929] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:23.154711 23783 log.cc:1199] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/4ea63540149c4c02abb8ae51209060a3
I20251023 20:15:23.155148 23783 ts_tablet_manager.cc:1950] T 4ea63540149c4c02abb8ae51209060a3 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:23.156002 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 502) took 1542 ms. Trace:
I20251023 20:15:23.156179 23783 rpcz_store.cc:276] 1023 20:15:21.613584 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.853942 (+240358us) service_pool.cc:225] Handling call
1023 20:15:23.117213 (+1263271us) ts_tablet_manager.cc:1151] Deleting tablet 4ea63540149c4c02abb8ae51209060a3
1023 20:15:23.117225 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.138121 (+ 20896us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.154294 (+ 16173us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.155941 (+ 1647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249281}
I20251023 20:15:23.157223 23786 tablet_bootstrap.cc:492] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.157193 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 504) took 1378 ms. Trace:
I20251023 20:15:23.157218 23787 tablet_bootstrap.cc:492] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.157307 23783 rpcz_store.cc:276] 1023 20:15:21.778769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.867956 (+ 89187us) service_pool.cc:225] Handling call
1023 20:15:23.156941 (+1288985us) ts_tablet_manager.cc:1151] Deleting tablet f103bece2fa240c19c23ac51046c13ed
1023 20:15:23.156953 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.157130 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288834}
I20251023 20:15:23.157485 23786 ts_tablet_manager.cc:1403] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.008s
I20251023 20:15:23.157913 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 501) took 1544 ms. Trace:
I20251023 20:15:23.158043 23783 rpcz_store.cc:276] 1023 20:15:21.613328 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.853420 (+240092us) service_pool.cc:225] Handling call
1023 20:15:23.157763 (+1304343us) ts_tablet_manager.cc:1151] Deleting tablet 1822fdf1f5f44dd4a742aa3c2f8443cf
1023 20:15:23.157774 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.157867 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288766}
I20251023 20:15:23.157436 23787 ts_tablet_manager.cc:1403] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.001s
I20251023 20:15:23.158370 23786 raft_consensus.cc:359] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.158591 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 507) took 1341 ms. Trace:
I20251023 20:15:23.158670 23786 raft_consensus.cc:385] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.158784 23786 raft_consensus.cc:740] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.158716 23783 rpcz_store.cc:276] 1023 20:15:21.816794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.961288 (+144494us) service_pool.cc:225] Handling call
1023 20:15:23.158448 (+1197160us) ts_tablet_manager.cc:1151] Deleting tablet a841ffdc6cdd432dac7afe2e2b53f224
1023 20:15:23.158456 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.158535 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1165422}
I20251023 20:15:23.159140 23786 consensus_queue.cc:260] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.159296 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 509) took 1342 ms. Trace:
I20251023 20:15:23.159720 23786 ts_tablet_manager.cc:1434] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:23.160255 23786 tablet_bootstrap.cc:492] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.159320 23787 raft_consensus.cc:359] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.162058 23787 raft_consensus.cc:385] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.162191 23787 raft_consensus.cc:740] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.162739 23786 tablet_bootstrap.cc:654] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.163589 24198 raft_consensus.cc:493] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.163720 24198 raft_consensus.cc:515] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.164357 24198 leader_election.cc:290] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.165043 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7301d6121c9d4c8dbc642e895aa4e9f6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.165479 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7301d6121c9d4c8dbc642e895aa4e9f6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:23.165915 23604 leader_election.cc:343] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:23.166266 23605 leader_election.cc:343] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.166389 23605 leader_election.cc:304] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.166843 24198 raft_consensus.cc:2749] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.167199 23790 tablet_bootstrap.cc:492] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.167354 23790 ts_tablet_manager.cc:1403] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.005s
I20251023 20:15:23.166239 23787 consensus_queue.cc:260] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.167912 23787 ts_tablet_manager.cc:1434] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:23.161774 23783 rpcz_store.cc:276] 1023 20:15:21.816987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.993065 (+176078us) service_pool.cc:225] Handling call
1023 20:15:23.159047 (+1165982us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:23.159055 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.159134 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160037}
I20251023 20:15:23.168324 23787 tablet_bootstrap.cc:492] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.168238 23790 raft_consensus.cc:359] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.168434 23790 raft_consensus.cc:385] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.168535 23790 raft_consensus.cc:740] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.168751 23790 consensus_queue.cc:260] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.170446 23790 ts_tablet_manager.cc:1434] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:23.170804 23790 tablet_bootstrap.cc:492] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.173331 23790 tablet_bootstrap.cc:654] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.177927 23787 tablet_bootstrap.cc:654] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.179080 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 510) took 1352 ms. Trace:
I20251023 20:15:23.179193 23783 rpcz_store.cc:276] 1023 20:15:21.826160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.999045 (+172885us) service_pool.cc:225] Handling call
1023 20:15:23.168085 (+1169040us) ts_tablet_manager.cc:1151] Deleting tablet a8eaa2167f82475ca2f6a110aba288ec
1023 20:15:23.178886 (+ 10801us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.179027 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168863}
I20251023 20:15:23.179664 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 508) took 1362 ms. Trace:
I20251023 20:15:23.179828 23783 rpcz_store.cc:276] 1023 20:15:21.816918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:21.964382 (+147464us) service_pool.cc:225] Handling call
1023 20:15:23.179462 (+1215080us) ts_tablet_manager.cc:1151] Deleting tablet 40c659c273c449939d0dd708d39b393f
1023 20:15:23.179472 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.179605 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167086}
I20251023 20:15:23.180197 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 513) took 1338 ms. Trace:
I20251023 20:15:23.180282 23783 rpcz_store.cc:276] 1023 20:15:21.841896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:22.011671 (+169775us) service_pool.cc:225] Handling call
1023 20:15:23.180060 (+1168389us) ts_tablet_manager.cc:1151] Deleting tablet e194d03a6aa946fbb2c79d91b196440b
1023 20:15:23.180070 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.180150 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1139727}
I20251023 20:15:23.180557 23783 tablet_replica.cc:333] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:23.180752 23783 raft_consensus.cc:2243] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:23.180936 23783 raft_consensus.cc:2272] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:23.181494 23783 ts_tablet_manager.cc:1916] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:23.184289 24186 raft_consensus.cc:493] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.184434 24186 raft_consensus.cc:515] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.185107 24186 leader_election.cc:290] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.186506 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90d58bceecba442898a3d742fa49a4a2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.186549 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90d58bceecba442898a3d742fa49a4a2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.186707 23670 raft_consensus.cc:2468] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.186745 23536 raft_consensus.cc:2468] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.187368 23333 leader_election.cc:304] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.188055 24186 raft_consensus.cc:2804] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.188162 24186 raft_consensus.cc:493] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.188288 24186 raft_consensus.cc:3060] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.191061 24186 raft_consensus.cc:515] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.192256 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90d58bceecba442898a3d742fa49a4a2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.192453 23670 raft_consensus.cc:3060] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.192701 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90d58bceecba442898a3d742fa49a4a2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.192909 23536 raft_consensus.cc:3060] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.195173 23670 raft_consensus.cc:2468] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.195801 23536 raft_consensus.cc:2468] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.196215 23333 leader_election.cc:304] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.196488 24230 raft_consensus.cc:2804] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.196662 24230 raft_consensus.cc:697] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.197013 24186 leader_election.cc:290] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.196995 24230 consensus_queue.cc:237] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.199219 23787 tablet_bootstrap.cc:492] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.199230 23790 tablet_bootstrap.cc:492] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.199388 23787 ts_tablet_manager.cc:1403] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20251023 20:15:23.199406 23790 ts_tablet_manager.cc:1403] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.007s
I20251023 20:15:23.199666 23786 tablet_bootstrap.cc:492] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.199862 23786 ts_tablet_manager.cc:1403] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.002s
I20251023 20:15:23.200198 23790 raft_consensus.cc:359] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.200139 23787 raft_consensus.cc:359] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.200393 23787 raft_consensus.cc:385] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.200402 23790 raft_consensus.cc:385] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.200476 23790 raft_consensus.cc:740] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.200538 23787 raft_consensus.cc:740] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.200733 23790 consensus_queue.cc:260] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.200841 23787 consensus_queue.cc:260] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.200930 23786 raft_consensus.cc:359] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.201570 23790 ts_tablet_manager.cc:1434] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251023 20:15:23.201570 23786 raft_consensus.cc:385] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.201663 23786 raft_consensus.cc:740] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.201915 23786 consensus_queue.cc:260] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.202566 23786 ts_tablet_manager.cc:1434] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:23.202970 23790 tablet_bootstrap.cc:492] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.205991 23786 tablet_bootstrap.cc:492] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.206476 23787 ts_tablet_manager.cc:1434] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:23.208937 23786 tablet_bootstrap.cc:654] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.211645 23787 tablet_bootstrap.cc:492] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.209478 24184 raft_consensus.cc:493] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.212481 24184 raft_consensus.cc:515] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.213270 24184 leader_election.cc:290] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.213742 23787 tablet_bootstrap.cc:654] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:23.214380 23471 leader_election.cc:343] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: e0f8d8184a68453abd2792ccc7740aa1
W20251023 20:15:23.215869 23470 leader_election.cc:343] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e0f8d8184a68453abd2792ccc7740aa1
I20251023 20:15:23.215999 23470 leader_election.cc:304] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.216254 24184 raft_consensus.cc:2749] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.217321 23790 tablet_bootstrap.cc:654] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:23.227906 23604 consensus_peers.cc:597] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 364e3b9a07e44c8992f6d9cbda6a86e5. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:23.231755 23783 ts_tablet_manager.cc:1929] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:23.232121 23790 tablet_bootstrap.cc:492] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.232290 23790 ts_tablet_manager.cc:1403] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20251023 20:15:23.232995 23783 log.cc:1199] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/cff337a6e728489b99fa8c5d4dfece2b
I20251023 20:15:23.232889 23787 tablet_bootstrap.cc:492] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.233309 23787 ts_tablet_manager.cc:1403] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.003s
I20251023 20:15:23.233254 23790 raft_consensus.cc:359] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.233433 23790 raft_consensus.cc:385] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.233526 23790 raft_consensus.cc:740] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.233820 23790 consensus_queue.cc:260] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.234357 23790 ts_tablet_manager.cc:1434] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.234722 23790 tablet_bootstrap.cc:492] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.236613 23790 tablet_bootstrap.cc:654] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.237622 23786 tablet_bootstrap.cc:492] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.237860 23786 ts_tablet_manager.cc:1403] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.003s
I20251023 20:15:23.238682 23786 raft_consensus.cc:359] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.238950 23786 raft_consensus.cc:385] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.239078 23786 raft_consensus.cc:740] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.239370 23786 consensus_queue.cc:260] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.240093 23786 ts_tablet_manager.cc:1434] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:23.240671 23787 raft_consensus.cc:359] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.240924 23787 raft_consensus.cc:385] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.241065 23787 raft_consensus.cc:740] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.241163 23786 tablet_bootstrap.cc:492] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.242967 23786 tablet_bootstrap.cc:654] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.254220 23786 tablet_bootstrap.cc:492] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.254467 23786 ts_tablet_manager.cc:1403] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251023 20:15:23.255295 23786 raft_consensus.cc:359] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.255560 23786 raft_consensus.cc:385] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.255681 23786 raft_consensus.cc:740] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.255955 23786 consensus_queue.cc:260] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.241389 23787 consensus_queue.cc:260] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.256693 23786 ts_tablet_manager.cc:1434] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.257364 23786 tablet_bootstrap.cc:492] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.259336 23787 ts_tablet_manager.cc:1434] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20251023 20:15:23.260224 23787 tablet_bootstrap.cc:492] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.260828 24184 raft_consensus.cc:493] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.261005 24184 raft_consensus.cc:515] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.261744 24184 leader_election.cc:290] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:23.262982 23470 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 585a9af140d9479fb83d13e68a62300c
W20251023 20:15:23.263444 23471 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 585a9af140d9479fb83d13e68a62300c
I20251023 20:15:23.263597 23471 leader_election.cc:304] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.264063 24184 raft_consensus.cc:2749] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.269798 23790 tablet_bootstrap.cc:492] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.269989 23790 ts_tablet_manager.cc:1403] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.012s
I20251023 20:15:23.270723 23790 raft_consensus.cc:359] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.270926 23790 raft_consensus.cc:385] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.270992 23790 raft_consensus.cc:740] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.271229 23790 consensus_queue.cc:260] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.271734 23790 ts_tablet_manager.cc:1434] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:23.272194 23786 tablet_bootstrap.cc:654] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.272539 23790 tablet_bootstrap.cc:492] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.274209 23790 tablet_bootstrap.cc:654] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.277493 23783 ts_tablet_manager.cc:1950] T cff337a6e728489b99fa8c5d4dfece2b P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:23.278373 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 515) took 1436 ms. Trace:
I20251023 20:15:23.278501 23783 rpcz_store.cc:276] 1023 20:15:21.842116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:22.026274 (+184158us) service_pool.cc:225] Handling call
1023 20:15:23.180478 (+1154204us) ts_tablet_manager.cc:1151] Deleting tablet cff337a6e728489b99fa8c5d4dfece2b
1023 20:15:23.180486 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.184729 (+ 4243us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.231601 (+ 46872us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:23.278331 (+ 46730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095073}
I20251023 20:15:23.279124 23787 tablet_bootstrap.cc:654] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.279548 23790 tablet_bootstrap.cc:492] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.279692 23790 ts_tablet_manager.cc:1403] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20251023 20:15:23.281112 23790 raft_consensus.cc:359] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.281288 23790 raft_consensus.cc:385] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.281371 23790 raft_consensus.cc:740] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.281395 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 525) took 1216 ms. Trace:
I20251023 20:15:23.281515 23783 rpcz_store.cc:276] 1023 20:15:22.064534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:22.068610 (+ 4076us) service_pool.cc:225] Handling call
1023 20:15:23.281228 (+1212618us) ts_tablet_manager.cc:1151] Deleting tablet db758a7a7a89410f8ac1da72048fbc56
1023 20:15:23.281238 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.281355 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059312}
I20251023 20:15:23.281615 23790 consensus_queue.cc:260] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.282264 23790 ts_tablet_manager.cc:1434] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251023 20:15:23.282903 23790 tablet_bootstrap.cc:492] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.284840 23790 tablet_bootstrap.cc:654] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.285326 23786 tablet_bootstrap.cc:492] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.285504 23786 ts_tablet_manager.cc:1403] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20251023 20:15:23.286309 23786 raft_consensus.cc:359] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.286499 23786 raft_consensus.cc:385] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.286571 23786 raft_consensus.cc:740] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.281883 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 524) took 1226 ms. Trace:
I20251023 20:15:23.286814 23786 consensus_queue.cc:260] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.286851 23783 rpcz_store.cc:276] 1023 20:15:22.063633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:22.098607 (+ 34974us) service_pool.cc:225] Handling call
1023 20:15:23.281753 (+1183146us) ts_tablet_manager.cc:1151] Deleting tablet fbab01ecb27a426188689408079eaa01
1023 20:15:23.281762 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.281854 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052047}
I20251023 20:15:23.287302 23786 ts_tablet_manager.cc:1434] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:23.287279 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 529) took 1062 ms. Trace:
I20251023 20:15:23.287619 23786 tablet_bootstrap.cc:492] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.290580 24184 raft_consensus.cc:493] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.290760 24184 raft_consensus.cc:515] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.291363 24184 leader_election.cc:290] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.292035 23786 tablet_bootstrap.cc:654] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:23.292590 23470 leader_election.cc:343] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e418a62d00824cff89b37a7bace5c469
I20251023 20:15:23.284230 23787 tablet_bootstrap.cc:492] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.292919 23787 ts_tablet_manager.cc:1403] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20251023 20:15:23.297938 24184 raft_consensus.cc:493] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.298158 24184 raft_consensus.cc:515] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.298826 24184 leader_election.cc:290] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:23.299687 23470 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:23.287387 23783 rpcz_store.cc:276] 1023 20:15:22.224522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:22.228888 (+ 4366us) service_pool.cc:225] Handling call
1023 20:15:23.287130 (+1058242us) ts_tablet_manager.cc:1151] Deleting tablet 01fa5f9c700b458ea147373d78551573
1023 20:15:23.287140 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.287247 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":983387}
I20251023 20:15:23.300371 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 530) took 1060 ms. Trace:
I20251023 20:15:23.300489 23783 rpcz_store.cc:276] 1023 20:15:22.239549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:22.240857 (+ 1308us) service_pool.cc:225] Handling call
1023 20:15:23.300168 (+1059311us) ts_tablet_manager.cc:1151] Deleting tablet 419e41d7a37f4b888cb242b089c28790
1023 20:15:23.300180 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.300337 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":991704}
I20251023 20:15:23.300841 23783 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38382 (request call id 531) took 1051 ms. Trace:
I20251023 20:15:23.300947 23783 rpcz_store.cc:276] 1023 20:15:22.249326 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:22.257859 (+ 8533us) service_pool.cc:225] Handling call
1023 20:15:23.300711 (+1042852us) ts_tablet_manager.cc:1151] Deleting tablet 370a790a053446b2b06314c7db242270
1023 20:15:23.300720 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:23.300797 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":957390}
I20251023 20:15:23.301435 23783 tablet_replica.cc:333] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:23.301496 23471 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:23.301528 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e418a62d00824cff89b37a7bace5c469" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.301697 23783 raft_consensus.cc:2243] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:23.301717 23471 leader_election.cc:304] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.301750 23397 raft_consensus.cc:2468] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:23.301882 23783 raft_consensus.cc:2272] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:23.302412 23783 ts_tablet_manager.cc:1916] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:23.302651 24259 raft_consensus.cc:2749] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.303635 23471 leader_election.cc:304] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.303897 24259 raft_consensus.cc:2804] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.304052 24259 raft_consensus.cc:493] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.304243 24259 raft_consensus.cc:3060] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.305722 23786 tablet_bootstrap.cc:492] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.305891 23786 ts_tablet_manager.cc:1403] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20251023 20:15:23.307282 23786 raft_consensus.cc:359] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.307475 23786 raft_consensus.cc:385] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.307571 23786 raft_consensus.cc:740] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.307808 23790 tablet_bootstrap.cc:492] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.307873 23786 consensus_queue.cc:260] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.307991 23790 ts_tablet_manager.cc:1403] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.007s
I20251023 20:15:23.308609 23786 ts_tablet_manager.cc:1434] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:23.308789 23790 raft_consensus.cc:359] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.308990 23790 raft_consensus.cc:385] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.309069 23790 raft_consensus.cc:740] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.309343 23786 tablet_bootstrap.cc:492] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.309314 23790 consensus_queue.cc:260] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.309953 23790 ts_tablet_manager.cc:1434] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:23.310282 23790 tablet_bootstrap.cc:492] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.312005 23790 tablet_bootstrap.cc:654] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.298736 23787 raft_consensus.cc:359] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.315215 23790 tablet_bootstrap.cc:492] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.315357 23790 ts_tablet_manager.cc:1403] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20251023 20:15:23.316807 23790 raft_consensus.cc:359] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.316987 23790 raft_consensus.cc:385] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.317073 23790 raft_consensus.cc:740] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.317296 23790 consensus_queue.cc:260] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.317943 23790 ts_tablet_manager.cc:1434] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:23.318198 23786 tablet_bootstrap.cc:654] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.318480 23790 tablet_bootstrap.cc:492] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.320118 23790 tablet_bootstrap.cc:654] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.320199 24259 raft_consensus.cc:515] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.320865 24259 leader_election.cc:290] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.321067 24262 raft_consensus.cc:493] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae5a310b4a514c0f812bda133c756062)
I20251023 20:15:23.321298 24262 raft_consensus.cc:515] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:23.323207 24013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "690ffde75cd745b6a87925967d54471a" candidate_term: 2 candidate_status { last_received { term: 1 index: 658 } } ignore_live_leader: false dest_uuid: "6545140d94b7485aa81c51ad233355cd" is_pre_election: true
I20251023 20:15:23.323556 24013 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 690ffde75cd745b6a87925967d54471a in term 1.
I20251023 20:15:23.324115 23205 leader_election.cc:304] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd, 690ffde75cd745b6a87925967d54471a; no voters:
I20251023 20:15:23.324820 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e418a62d00824cff89b37a7bace5c469" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:23.325018 23397 raft_consensus.cc:3060] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.328411 23397 raft_consensus.cc:2468] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:23.331781 23471 leader_election.cc:304] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.332202 24259 raft_consensus.cc:2804] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.332638 24259 raft_consensus.cc:697] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:23.332774 24265 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:15:23.333006 24265 raft_consensus.cc:493] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae5a310b4a514c0f812bda133c756062)
I20251023 20:15:23.333144 24262 leader_election.cc:290] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331), ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509)
I20251023 20:15:23.333283 24265 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:23.333303 24259 consensus_queue.cc:237] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:23.335016 23470 leader_election.cc:343] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e418a62d00824cff89b37a7bace5c469
I20251023 20:15:23.334640 23787 raft_consensus.cc:385] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:23.335614 23605 consensus_peers.cc:597] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 -> Peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Couldn't send request to peer d220aba7c9c0400398ffde917376ee48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 364e3b9a07e44c8992f6d9cbda6a86e5. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:23.335893 23787 raft_consensus.cc:740] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.336261 23787 consensus_queue.cc:260] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.338758 23787 ts_tablet_manager.cc:1434] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20251023 20:15:23.339169 23787 tablet_bootstrap.cc:492] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.339373 23790 tablet_bootstrap.cc:492] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.339643 23790 ts_tablet_manager.cc:1403] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20251023 20:15:23.341753 23790 raft_consensus.cc:359] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.341944 23790 raft_consensus.cc:385] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.342016 23790 raft_consensus.cc:740] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.342339 23790 consensus_queue.cc:260] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.343041 23790 ts_tablet_manager.cc:1434] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:23.343786 23790 tablet_bootstrap.cc:492] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.345515 23787 tablet_bootstrap.cc:654] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.345949 23790 tablet_bootstrap.cc:654] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.354077 23790 tablet_bootstrap.cc:492] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.354234 23790 ts_tablet_manager.cc:1403] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20251023 20:15:23.355625 23790 raft_consensus.cc:359] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.355839 23790 raft_consensus.cc:385] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.355924 23790 raft_consensus.cc:740] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.356184 23790 consensus_queue.cc:260] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.356859 23790 ts_tablet_manager.cc:1434] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:23.357148 23787 tablet_bootstrap.cc:492] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.357242 23790 tablet_bootstrap.cc:492] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.357471 23787 ts_tablet_manager.cc:1403] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20251023 20:15:23.358300 23787 raft_consensus.cc:359] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.358528 23787 raft_consensus.cc:385] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.358640 23787 raft_consensus.cc:740] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.358937 23790 tablet_bootstrap.cc:654] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.358925 23787 consensus_queue.cc:260] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.359643 23787 ts_tablet_manager.cc:1434] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:23.360000 23787 tablet_bootstrap.cc:492] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.361310 23783 ts_tablet_manager.cc:1929] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:23.361490 23783 log.cc:1199] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/7301d6121c9d4c8dbc642e895aa4e9f6
I20251023 20:15:23.361958 23783 ts_tablet_manager.cc:1950] T 7301d6121c9d4c8dbc642e895aa4e9f6 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:23.364847 23783 tablet_replica.cc:333] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:23.365134 23783 raft_consensus.cc:2243] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:23.356393 24265 raft_consensus.cc:515] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:23.365537 23783 raft_consensus.cc:2272] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:23.371946 23786 tablet_bootstrap.cc:492] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.372118 23786 ts_tablet_manager.cc:1403] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.010s
I20251023 20:15:23.372213 23783 ts_tablet_manager.cc:1916] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:23.372915 23786 raft_consensus.cc:359] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.373137 23786 raft_consensus.cc:385] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.373204 23786 raft_consensus.cc:740] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.373384 24013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "690ffde75cd745b6a87925967d54471a" candidate_term: 2 candidate_status { last_received { term: 1 index: 658 } } ignore_live_leader: false dest_uuid: "6545140d94b7485aa81c51ad233355cd"
I20251023 20:15:23.373454 23786 consensus_queue.cc:260] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.373611 24013 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:23.373664 24265 leader_election.cc:290] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [CANDIDATE]: Term 2 election: Requested vote from peers 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331), ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509)
I20251023 20:15:23.373973 23786 ts_tablet_manager.cc:1434] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:23.374315 23786 tablet_bootstrap.cc:492] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.376142 23786 tablet_bootstrap.cc:654] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.380199 23787 tablet_bootstrap.cc:654] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.380708 24013 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 690ffde75cd745b6a87925967d54471a in term 2.
I20251023 20:15:23.382076 23205 leader_election.cc:304] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd, 690ffde75cd745b6a87925967d54471a; no voters:
I20251023 20:15:23.382596 23790 tablet_bootstrap.cc:492] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.382750 24265 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:15:23.382803 23790 ts_tablet_manager.cc:1403] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20251023 20:15:23.383114 24265 raft_consensus.cc:697] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [term 2 LEADER]: Becoming Leader. State: Replica: 690ffde75cd745b6a87925967d54471a, State: Running, Role: LEADER
I20251023 20:15:23.383670 24265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 658, Committed index: 658, Last appended: 1.658, Last appended by leader: 658, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:23.385262 23790 raft_consensus.cc:359] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.385471 23790 raft_consensus.cc:385] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.385541 23790 raft_consensus.cc:740] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.385787 23790 consensus_queue.cc:260] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.385906 24262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690ffde75cd745b6a87925967d54471a. Latest consensus state: current_term: 2 leader_uuid: "690ffde75cd745b6a87925967d54471a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:23.386078 24262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:23.386314 23790 ts_tablet_manager.cc:1434] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:23.388105 24230 raft_consensus.cc:493] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.388240 24230 raft_consensus.cc:515] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.388844 24230 leader_election.cc:290] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.389948 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fc7e18054c440098dc2470143ea7fae" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.390166 23536 raft_consensus.cc:2468] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:23.390705 23333 leader_election.cc:343] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 9fc7e18054c440098dc2470143ea7fae
I20251023 20:15:23.391242 23333 leader_election.cc:304] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.391705 24230 raft_consensus.cc:2804] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.392311 24230 raft_consensus.cc:493] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.392411 24230 raft_consensus.cc:3060] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.393700 23790 tablet_bootstrap.cc:492] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.394651 24259 raft_consensus.cc:493] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.394805 24259 raft_consensus.cc:515] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.395447 23790 tablet_bootstrap.cc:654] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.395426 24259 leader_election.cc:290] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:23.396451 23470 leader_election.cc:343] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: c95db7b5b9fd46acae8ffa7119915907
W20251023 20:15:23.396637 23471 leader_election.cc:343] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: c95db7b5b9fd46acae8ffa7119915907
I20251023 20:15:23.396772 23471 leader_election.cc:304] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.397320 24259 raft_consensus.cc:2749] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.397771 23786 tablet_bootstrap.cc:492] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.397922 23786 ts_tablet_manager.cc:1403] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.003s
I20251023 20:15:23.398190 24230 raft_consensus.cc:515] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.398667 23786 raft_consensus.cc:359] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.398846 23786 raft_consensus.cc:385] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.398924 23786 raft_consensus.cc:740] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.399121 23786 consensus_queue.cc:260] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.401193 23786 ts_tablet_manager.cc:1434] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251023 20:15:23.401635 23333 leader_election.cc:343] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 9fc7e18054c440098dc2470143ea7fae
I20251023 20:15:23.401918 23786 tablet_bootstrap.cc:492] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.402328 24272 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:23.404289 23786 tablet_bootstrap.cc:654] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.405264 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fc7e18054c440098dc2470143ea7fae" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.405475 23536 raft_consensus.cc:3060] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.406780 23787 tablet_bootstrap.cc:492] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.406870 24259 raft_consensus.cc:493] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.407055 23787 ts_tablet_manager.cc:1403] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.000s
I20251023 20:15:23.407037 24259 raft_consensus.cc:515] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.407665 24259 leader_election.cc:290] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.408286 23536 raft_consensus.cc:2468] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:23.408365 23470 leader_election.cc:343] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 8da82e6d24b14a129b91bb5298139d07
I20251023 20:15:23.408831 23787 raft_consensus.cc:359] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.409263 23787 raft_consensus.cc:385] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.409274 23333 leader_election.cc:304] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.409461 23787 raft_consensus.cc:740] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.409859 24220 raft_consensus.cc:2804] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.409895 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8da82e6d24b14a129b91bb5298139d07" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.410012 24220 raft_consensus.cc:697] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.410092 23397 raft_consensus.cc:2468] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:23.410279 24220 consensus_queue.cc:237] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.411705 23787 consensus_queue.cc:260] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.412508 23787 ts_tablet_manager.cc:1434] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:23.413039 23471 leader_election.cc:304] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.413564 24259 raft_consensus.cc:2804] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.413661 24259 raft_consensus.cc:493] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.413770 24259 raft_consensus.cc:3060] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.415066 23783 ts_tablet_manager.cc:1929] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:23.415211 23783 log.cc:1199] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/364e3b9a07e44c8992f6d9cbda6a86e5
I20251023 20:15:23.415593 23783 ts_tablet_manager.cc:1950] T 364e3b9a07e44c8992f6d9cbda6a86e5 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:23.416164 23787 tablet_bootstrap.cc:492] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.420011 23787 tablet_bootstrap.cc:654] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.427398 24259 raft_consensus.cc:515] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.428599 24230 leader_election.cc:290] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.431950 24184 raft_consensus.cc:493] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.434360 24184 raft_consensus.cc:515] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.435010 24184 leader_election.cc:290] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:23.435536 24259 leader_election.cc:290] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.436172 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8da82e6d24b14a129b91bb5298139d07" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:23.436416 23399 raft_consensus.cc:3060] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.439249 23399 raft_consensus.cc:2468] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:23.439494 23471 leader_election.cc:343] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 6558265ff9e249c68123577958f1401a
W20251023 20:15:23.439679 23470 leader_election.cc:343] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 6558265ff9e249c68123577958f1401a
I20251023 20:15:23.439792 23470 leader_election.cc:304] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.440039 23471 leader_election.cc:304] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:23.440218 23470 leader_election.cc:343] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 8da82e6d24b14a129b91bb5298139d07
I20251023 20:15:23.440331 24184 raft_consensus.cc:2749] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.440511 23790 tablet_bootstrap.cc:492] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.440546 24184 raft_consensus.cc:2804] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.440670 23790 ts_tablet_manager.cc:1403] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.047s user 0.022s sys 0.014s
I20251023 20:15:23.440665 24184 raft_consensus.cc:697] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:23.440935 24184 consensus_queue.cc:237] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.441491 23790 raft_consensus.cc:359] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.441691 23790 raft_consensus.cc:385] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.441787 23790 raft_consensus.cc:740] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.442040 23790 consensus_queue.cc:260] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.442718 23790 ts_tablet_manager.cc:1434] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.443038 23790 tablet_bootstrap.cc:492] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.445614 23783 tablet_replica.cc:333] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:23.445652 23790 tablet_bootstrap.cc:654] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.445856 23783 raft_consensus.cc:2243] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:23.448359 23786 tablet_bootstrap.cc:492] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.448550 23786 ts_tablet_manager.cc:1403] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.013s
I20251023 20:15:23.449404 23786 raft_consensus.cc:359] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.449602 23786 raft_consensus.cc:385] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.449687 23786 raft_consensus.cc:740] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.449911 23786 consensus_queue.cc:260] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.450627 23786 ts_tablet_manager.cc:1434] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.450953 23786 tablet_bootstrap.cc:492] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.452773 23786 tablet_bootstrap.cc:654] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.457933 23786 tablet_bootstrap.cc:492] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.458179 23786 ts_tablet_manager.cc:1403] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20251023 20:15:23.459048 23786 raft_consensus.cc:359] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.459232 23786 raft_consensus.cc:385] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.459309 23786 raft_consensus.cc:740] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.459555 23786 consensus_queue.cc:260] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.460266 23786 ts_tablet_manager.cc:1434] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.460572 23786 tablet_bootstrap.cc:492] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.446035 23783 raft_consensus.cc:2272] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:23.461783 23783 ts_tablet_manager.cc:1916] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:23.462838 23786 tablet_bootstrap.cc:654] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.464701 24174 raft_consensus.cc:493] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.464893 24174 raft_consensus.cc:515] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.465597 23787 tablet_bootstrap.cc:492] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.466250 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4778436aed64b53adb2bab54bcf263b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.466694 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4778436aed64b53adb2bab54bcf263b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.466909 23399 raft_consensus.cc:2468] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:23.467170 23790 tablet_bootstrap.cc:492] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.467350 23790 ts_tablet_manager.cc:1403] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.011s
I20251023 20:15:23.465655 24174 leader_election.cc:290] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:23.466898 23604 leader_election.cc:343] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.468752 23787 ts_tablet_manager.cc:1403] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20251023 20:15:23.468861 23790 raft_consensus.cc:359] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.469072 23790 raft_consensus.cc:385] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.468705 23605 leader_election.cc:304] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.469170 23790 raft_consensus.cc:740] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.469472 23790 consensus_queue.cc:260] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.469528 24174 raft_consensus.cc:2804] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.469640 24174 raft_consensus.cc:493] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.469738 24174 raft_consensus.cc:3060] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.470293 23787 raft_consensus.cc:359] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.470669 23787 raft_consensus.cc:385] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.470824 23787 raft_consensus.cc:740] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.471141 23787 consensus_queue.cc:260] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.471717 23787 ts_tablet_manager.cc:1434] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.472072 23787 tablet_bootstrap.cc:492] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.472474 24174 raft_consensus.cc:515] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.474014 23787 tablet_bootstrap.cc:654] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.475680 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4778436aed64b53adb2bab54bcf263b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.476418 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4778436aed64b53adb2bab54bcf263b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:23.476624 23399 raft_consensus.cc:3060] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.477766 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_baf578c1b5e748e09f244f10685992a8 [id=006294844eac4bcabdd6508738bdc4cf]
I20251023 20:15:23.478948 23399 raft_consensus.cc:2468] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:23.475459 24174 leader_election.cc:290] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.476032 24160 raft_consensus.cc:493] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:23.476241 23604 leader_election.cc:343] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.479655 23790 ts_tablet_manager.cc:1434] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251023 20:15:23.479725 24160 raft_consensus.cc:515] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.480015 23790 tablet_bootstrap.cc:492] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.480350 24160 leader_election.cc:290] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.481696 23605 leader_election.cc:304] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.481972 24160 raft_consensus.cc:2804] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.482159 23790 tablet_bootstrap.cc:654] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.482142 24160 raft_consensus.cc:697] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:23.482395 24160 consensus_queue.cc:237] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.482970 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ae19cc80194363b8a9c96fbafb4ebe" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:23.483563 23604 leader_election.cc:343] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.483775 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ae19cc80194363b8a9c96fbafb4ebe" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:23.485514 23605 leader_election.cc:343] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.485648 23605 leader_election.cc:304] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.485998 24174 raft_consensus.cc:2749] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.486318 23787 tablet_bootstrap.cc:492] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.486317 24174 raft_consensus.cc:493] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.486586 24174 raft_consensus.cc:515] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.486670 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]
I20251023 20:15:23.487198 24174 leader_election.cc:290] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.487673 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.488109 23787 ts_tablet_manager.cc:1403] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
W20251023 20:15:23.488139 23604 leader_election.cc:343] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.488965 23787 raft_consensus.cc:359] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.489233 23787 raft_consensus.cc:385] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.489355 23787 raft_consensus.cc:740] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.489313 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:23.489986 23605 leader_election.cc:343] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.490134 23605 leader_election.cc:304] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.490417 23783 ts_tablet_manager.cc:1929] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:23.490535 24174 raft_consensus.cc:2749] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.490965 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_4c838e5585b948b9ba710086a76a2f3f [id=04d988798f7f4b56be2fe5ed680539d4]
I20251023 20:15:23.490981 23787 consensus_queue.cc:260] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.491631 23787 ts_tablet_manager.cc:1434] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:23.491737 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_30b62bd8bf6147a7a5800584576db269 [id=054cce203d7e4437a8a7aa22df487453]
I20251023 20:15:23.491986 23787 tablet_bootstrap.cc:492] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.492987 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]
I20251023 20:15:23.494714 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_934c5152009843bcbf11132bbe4c8ed2 [id=09f8335a5b03409c90559c8313f5d135]
I20251023 20:15:23.495410 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_1ec08f875c4f4a438cd2081db4326eb3 [id=0aefe460f19449a08ef7574681b2af94]
I20251023 20:15:23.490561 23783 log.cc:1199] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/52bc660b9da54c5d984381a267e1efa2
I20251023 20:15:23.497073 23783 ts_tablet_manager.cc:1950] T 52bc660b9da54c5d984381a267e1efa2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:23.498378 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e851810007344c51aa7a54934d6ad579 [id=0ddefd3e67364bc8bf23971811893894]
I20251023 20:15:23.498466 23786 tablet_bootstrap.cc:492] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.498845 23786 ts_tablet_manager.cc:1403] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.007s
I20251023 20:15:23.500942 23786 raft_consensus.cc:359] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.501155 23786 raft_consensus.cc:385] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.501262 23786 raft_consensus.cc:740] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.502027 23786 consensus_queue.cc:260] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.502550 23790 tablet_bootstrap.cc:492] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.502728 23786 ts_tablet_manager.cc:1434] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.502750 23790 ts_tablet_manager.cc:1403] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.005s
I20251023 20:15:23.503191 23786 tablet_bootstrap.cc:492] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.503647 23790 raft_consensus.cc:359] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.503849 23790 raft_consensus.cc:385] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.503937 23790 raft_consensus.cc:740] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.504194 23790 consensus_queue.cc:260] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.506654 23790 ts_tablet_manager.cc:1434] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:23.506968 23790 tablet_bootstrap.cc:492] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.508776 23790 tablet_bootstrap.cc:654] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.500111 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]
I20251023 20:15:23.511408 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_50879e22af484b719900c9b120534de0 [id=1245153d2d05472a9f20bc69214ca3ce]
I20251023 20:15:23.511914 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]
I20251023 20:15:23.512876 23786 tablet_bootstrap.cc:654] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.513417 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_365837ad05624d75855461852362db1c [id=18a582219c6248adac2c8e7e2611808d]
I20251023 20:15:23.514353 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_4fb0e33f75594ca59d2ecb391873d804 [id=1a32884612a34df48b257e1a18bb5e73]
I20251023 20:15:23.515421 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_1b20af8a893a4715affda818cffeca13 [id=201b60a519df46fcb92de45acff546cc]
I20251023 20:15:23.516142 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_3a5e9b90e86340ed80fdb4a8679777e6 [id=22d88af65ebd43389f6e58573143816c]
I20251023 20:15:23.516861 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_25427c3d36fb4ba4a01ac362787c81b8 [id=248b68f25e64412ba27e4c74b27c59d8]
I20251023 20:15:23.517897 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_2496fe6d77e44734bfced35dee7d6596 [id=24a5b1c93ab84b2984243fc1a7f41e0c]
I20251023 20:15:23.519141 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]
I20251023 20:15:23.520630 23787 tablet_bootstrap.cc:654] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.520816 23786 tablet_bootstrap.cc:492] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.520979 23786 ts_tablet_manager.cc:1403] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.008s
I20251023 20:15:23.521386 24184 raft_consensus.cc:493] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.521552 24184 raft_consensus.cc:515] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.522186 24184 leader_election.cc:290] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.522639 24160 raft_consensus.cc:493] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.522795 24160 raft_consensus.cc:515] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.522861 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215e21ca449c481d8716783a65105101" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.522043 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]
I20251023 20:15:23.523033 23670 raft_consensus.cc:2468] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:23.523437 24160 leader_election.cc:290] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.523639 23470 leader_election.cc:304] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.524103 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215e21ca449c481d8716783a65105101" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.524103 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2dbb7997d0f4e6bac48cfb196f5ccb6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.524369 23397 raft_consensus.cc:2468] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:23.524376 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2dbb7997d0f4e6bac48cfb196f5ccb6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.524616 24184 raft_consensus.cc:2804] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.524708 24184 raft_consensus.cc:493] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.524858 24184 raft_consensus.cc:3060] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.524780 23605 leader_election.cc:304] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.524852 23399 raft_consensus.cc:2468] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
W20251023 20:15:23.525394 23604 leader_election.cc:343] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.526038 23790 tablet_bootstrap.cc:492] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.526223 23790 ts_tablet_manager.cc:1403] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20251023 20:15:23.527107 23790 raft_consensus.cc:359] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.527335 23790 raft_consensus.cc:385] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.527419 23790 raft_consensus.cc:740] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.527633 24184 raft_consensus.cc:515] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.527719 23790 consensus_queue.cc:260] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.528313 23790 ts_tablet_manager.cc:1434] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.528766 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_7908a191ee99400093d9f2deb05e52d2 [id=2ad9425327fb485d89c61e102c983b30]
I20251023 20:15:23.529034 24160 raft_consensus.cc:493] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.529110 24174 raft_consensus.cc:2804] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.529277 24174 raft_consensus.cc:493] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.529296 24160 raft_consensus.cc:515] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.529418 24174 raft_consensus.cc:3060] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.529477 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d491cd3951344335be8170dcdcb94c4c [id=2c61abbba33e4ed4805127246e3cd0ba]
I20251023 20:15:23.529974 24160 leader_election.cc:290] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.530449 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e76f2e26dcd4cdfbf72de899f5a1458" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.530658 23399 raft_consensus.cc:2468] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:23.531096 23605 leader_election.cc:304] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.531340 23790 tablet_bootstrap.cc:492] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.531649 24160 raft_consensus.cc:2804] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.531746 24160 raft_consensus.cc:493] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.531843 24160 raft_consensus.cc:3060] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.532076 24174 raft_consensus.cc:515] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.532702 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e76f2e26dcd4cdfbf72de899f5a1458" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.532900 23536 raft_consensus.cc:2468] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:23.533710 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_4f8238f0492b4364b731c0bf5e2bd53a [id=2db1100447a44e7fb43e87cdf7247d14]
I20251023 20:15:23.534061 23790 tablet_bootstrap.cc:654] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.535019 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ec09021e59f8464f81cb6ceda1f1c567 [id=2fd818d424a44efdab6cf6fda7537ee8]
I20251023 20:15:23.535211 24160 raft_consensus.cc:515] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.536224 24184 leader_election.cc:290] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.536927 24160 leader_election.cc:290] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.536978 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215e21ca449c481d8716783a65105101" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.537186 23670 raft_consensus.cc:3060] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.537405 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]
I20251023 20:15:23.537869 23786 raft_consensus.cc:359] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.538048 23786 raft_consensus.cc:385] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.538138 23786 raft_consensus.cc:740] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.538149 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_87caa0c69120413a92c6b65337baad14 [id=3227f5fd7d78482c99fb9f654ba8e3c6]
I20251023 20:15:23.538420 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e76f2e26dcd4cdfbf72de899f5a1458" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.538429 23786 consensus_queue.cc:260] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.538630 23536 raft_consensus.cc:3060] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.538774 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]
I20251023 20:15:23.539086 23786 ts_tablet_manager.cc:1434] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251023 20:15:23.539178 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]
I20251023 20:15:23.539394 23786 tablet_bootstrap.cc:492] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.539572 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_191b94017cca4cfb946f38ab19ebfe20 [id=330c829b79fd4cce881fc565a3463863]
I20251023 20:15:23.539776 23670 raft_consensus.cc:2468] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:23.540153 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "215e21ca449c481d8716783a65105101" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:23.540270 23470 leader_election.cc:304] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.540359 23399 raft_consensus.cc:3060] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.540508 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]
I20251023 20:15:23.540546 24184 raft_consensus.cc:2804] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.540776 24184 raft_consensus.cc:697] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:23.540930 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b9e2bcb23af94936a9fbfd0817d9a51e [id=3528c0390b7240e8a565945035555dfa]
I20251023 20:15:23.541021 24184 consensus_queue.cc:237] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.541872 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_39fdd0d19cae4e928fdee07e74900b4b [id=356410224c0f44bdacbaa543e9dee27d]
I20251023 20:15:23.542243 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 [id=3973b56caa79477f8542118421741084]
I20251023 20:15:23.542953 23786 tablet_bootstrap.cc:654] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.543002 23399 raft_consensus.cc:2468] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:23.543454 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]
I20251023 20:15:23.543659 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e76f2e26dcd4cdfbf72de899f5a1458" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:23.544585 23397 raft_consensus.cc:3060] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.544864 23536 raft_consensus.cc:2468] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:23.545316 23604 leader_election.cc:304] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.545625 24174 leader_election.cc:290] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.545644 24160 raft_consensus.cc:2804] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.545826 24160 raft_consensus.cc:697] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:23.546231 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2dbb7997d0f4e6bac48cfb196f5ccb6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.546317 24160 consensus_queue.cc:237] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:23.546761 23604 leader_election.cc:343] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.546420 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_8dc2f44cc1fc43e4a9851a0587a74edb [id=3bf5870046244dcd817441dce750a1e2]
I20251023 20:15:23.547266 23397 raft_consensus.cc:2468] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:23.547356 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ac4b704784f2423a82ad591bbed856ab [id=3cf9ba4080ce4e5ead110ed71e43ce1f]
I20251023 20:15:23.548019 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2dbb7997d0f4e6bac48cfb196f5ccb6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:23.548341 23397 raft_consensus.cc:3060] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.548820 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_698f8853c4774ccdae2e37d28f90f368 [id=3e597b14c5d944838c32914dc1779d20]
I20251023 20:15:23.550088 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_7824bcf5ff034be78ed7dee87e4cd78d [id=44239d72fdcd48258921b9490c822137]
I20251023 20:15:23.551260 23397 raft_consensus.cc:2468] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:23.551681 23605 leader_election.cc:304] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.552146 24174 raft_consensus.cc:2804] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.552317 24174 raft_consensus.cc:697] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:23.552621 24174 consensus_queue.cc:237] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.552812 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_240b16ca6e1543f2aab607c1d6a447fd [id=4aa2d573bbb34a3ebda2ac6376a387ea]
I20251023 20:15:23.553642 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]
I20251023 20:15:23.554873 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]
I20251023 20:15:23.556107 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_6443f81020254f45a1b7fd93ad217e09 [id=4d43a0eda13041689a414ecbadb85e09]
I20251023 20:15:23.558693 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_baf8abd9a132412a8b53ac8d9e044dcf [id=51ea9f032fdd40bb9f0ea42bb80c5cbf]
I20251023 20:15:23.559945 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]
I20251023 20:15:23.562036 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b [id=58cb342d7d6142858153064d66e3712c]
I20251023 20:15:23.563656 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_de4427f05119413b8983f4e818072f98 [id=591b99acc8dd4e7388acb07fdf88a50a]
I20251023 20:15:23.564779 23786 tablet_bootstrap.cc:492] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.564954 23786 ts_tablet_manager.cc:1403] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20251023 20:15:23.565555 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d [id=5a207d264756433f94af5164462692ff]
I20251023 20:15:23.565748 23786 raft_consensus.cc:359] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.565968 23786 raft_consensus.cc:385] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.566040 23786 raft_consensus.cc:740] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.566423 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]
I20251023 20:15:23.568145 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]
I20251023 20:15:23.570297 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485]
I20251023 20:15:23.570955 24220 raft_consensus.cc:493] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.571152 24220 raft_consensus.cc:515] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.571863 24230 raft_consensus.cc:493] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.571972 24220 leader_election.cc:290] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.572029 24230 raft_consensus.cc:515] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.571899 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]
I20251023 20:15:23.572628 24230 leader_election.cc:290] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.572847 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dafb9db710f94c328186e3179bf0b319" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.573105 23536 raft_consensus.cc:2468] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.573064 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_9cac41d47a584e59a3c3e6fd69f9ad79 [id=6b26d29083ab46a29f00f2552ef20892]
I20251023 20:15:23.573625 23786 consensus_queue.cc:260] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.573899 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcbacd8e23bc4b129757d9faf89ce5b4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.574108 23670 raft_consensus.cc:2468] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.574445 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]
I20251023 20:15:23.574515 23333 leader_election.cc:304] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:23.574857 23333 leader_election.cc:343] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319
I20251023 20:15:23.575031 23786 ts_tablet_manager.cc:1434] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:23.575362 23786 tablet_bootstrap.cc:492] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.575444 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcbacd8e23bc4b129757d9faf89ce5b4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.575645 23536 raft_consensus.cc:2468] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.575768 23790 tablet_bootstrap.cc:492] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.575922 23790 ts_tablet_manager.cc:1403] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.017s
I20251023 20:15:23.576135 24230 raft_consensus.cc:2804] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.576244 24230 raft_consensus.cc:493] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.576328 24230 raft_consensus.cc:3060] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.576768 23790 raft_consensus.cc:359] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.576941 23790 raft_consensus.cc:385] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.577029 23790 raft_consensus.cc:740] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.577057 23786 tablet_bootstrap.cc:654] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.577800 23790 consensus_queue.cc:260] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.578454 23790 ts_tablet_manager.cc:1434] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:23.578998 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 [id=6defcb8665844b1d958c71d4fc34263a]
I20251023 20:15:23.579151 23786 tablet_bootstrap.cc:492] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.579282 23786 ts_tablet_manager.cc:1403] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251023 20:15:23.579669 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12 [id=6eda60ed5ff344f3a610b8139aa5a6f5]
I20251023 20:15:23.580076 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_efda7d26e2124ec4b1eeca32fe1b545e [id=6f9ad1c9f64a44a4b856013bb8fd4bd1]
I20251023 20:15:23.580039 23786 raft_consensus.cc:359] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.580214 23786 raft_consensus.cc:385] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.580304 23786 raft_consensus.cc:740] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.580552 23786 consensus_queue.cc:260] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.580650 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_37c825c402604e08a862951e693de880 [id=6fdeeacf0d0949838d6062e9330e4524]
I20251023 20:15:23.581229 23786 ts_tablet_manager.cc:1434] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.581281 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]
I20251023 20:15:23.581561 23786 tablet_bootstrap.cc:492] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.581665 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_f5ea72074b174da28b3738378167f49c [id=70e0efabed5a4dfe98eb7201a2b4aba2]
I20251023 20:15:23.582509 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_5e48fffc45314e45a6f7ac8bba5ab31d [id=7260df2ce8034b73956131dd07502d7b]
I20251023 20:15:23.583346 23786 tablet_bootstrap.cc:654] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.583732 23790 tablet_bootstrap.cc:492] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.585532 23790 tablet_bootstrap.cc:654] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.586344 24220 raft_consensus.cc:493] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.586534 23333 leader_election.cc:304] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.586495 24220 raft_consensus.cc:515] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.587060 24186 raft_consensus.cc:2804] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.587164 24186 raft_consensus.cc:493] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.587215 24220 leader_election.cc:290] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.587282 24186 raft_consensus.cc:3060] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.589593 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6 [id=77ff0c7e54e94defabffdbc82dbb609d]
I20251023 20:15:23.590080 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_3ad1b2da425540ccbe8b55d22f379b4a [id=7810e4d5193c4559891f1603fe361e11]
I20251023 20:15:23.590183 24186 raft_consensus.cc:515] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.591310 24186 leader_election.cc:290] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.590731 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998]
I20251023 20:15:23.592468 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcbacd8e23bc4b129757d9faf89ce5b4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.592701 23533 raft_consensus.cc:3060] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.593127 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 [id=79163b61e42a43d8879399cd5f31ce44]
I20251023 20:15:23.593531 23786 tablet_bootstrap.cc:492] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.593746 23786 ts_tablet_manager.cc:1403] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20251023 20:15:23.594249 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_6de6ff0130d74ad894cfa487113e1b19 [id=7a4098cc9d4e49a39ef792883f85d564]
I20251023 20:15:23.594601 23786 raft_consensus.cc:359] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.594815 23786 raft_consensus.cc:385] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.594892 23786 raft_consensus.cc:740] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.595180 23786 consensus_queue.cc:260] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.595300 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]
I20251023 20:15:23.595763 23786 ts_tablet_manager.cc:1434] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.596009 23533 raft_consensus.cc:2468] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.596077 23786 tablet_bootstrap.cc:492] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.596557 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_a5331f0f356d4ed88cf6787eb94db083 [id=7bf105aabdc944afac7c5d1e5afb824b]
I20251023 20:15:23.597452 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]
I20251023 20:15:23.597625 23786 tablet_bootstrap.cc:654] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.599316 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_775428d3af964d538226c836574a899d [id=818f0fca06904f09ad46c9e184c982b4]
I20251023 20:15:23.600533 23786 tablet_bootstrap.cc:492] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.600672 23786 ts_tablet_manager.cc:1403] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251023 20:15:23.601327 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.601414 23786 raft_consensus.cc:359] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.601589 23786 raft_consensus.cc:385] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.601567 23670 raft_consensus.cc:2468] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.601675 23786 raft_consensus.cc:740] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.601895 23786 consensus_queue.cc:260] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.602330 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]
I20251023 20:15:23.602524 23786 ts_tablet_manager.cc:1434] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.602917 23786 tablet_bootstrap.cc:492] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.603262 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_464dc629d50b4995b30ef5c463e9f316 [id=8477c10fb2f4422996d4eab8717d7fe8]
I20251023 20:15:23.604084 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]
I20251023 20:15:23.604664 23786 tablet_bootstrap.cc:654] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.605629 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_c760c475c8554c89aeb6fd77fd422797 [id=886ab40a90ab4410aa13fabf17a1ea91]
I20251023 20:15:23.606973 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]
I20251023 20:15:23.607108 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.607949 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]
I20251023 20:15:23.609016 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcbacd8e23bc4b129757d9faf89ce5b4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.609268 23670 raft_consensus.cc:3060] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.609521 23333 leader_election.cc:304] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.609903 23333 leader_election.cc:304] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.610338 24220 raft_consensus.cc:493] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.610513 24220 raft_consensus.cc:515] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.610944 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]
I20251023 20:15:23.611181 24220 leader_election.cc:290] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.611354 24220 raft_consensus.cc:2804] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.611433 24220 raft_consensus.cc:493] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.611522 24220 raft_consensus.cc:3060] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.611600 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]
I20251023 20:15:23.612227 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e6e2be420a6342bc9d1311bd4fe6080c [id=8eeb69b1cebc42dcaae9f3306aa18013]
I20251023 20:15:23.612377 23670 raft_consensus.cc:2468] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.612798 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b [id=905ffd6fc69a44c39b12a7efecc70fd4]
I20251023 20:15:23.613305 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_31442b53b2f748d0895b0014aa8ae96f [id=90d69f76e2324f5ea1ea4fee2b9042a6]
I20251023 20:15:23.617069 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]
I20251023 20:15:23.618079 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b98d2ec04db54e72a067731a38d21a56 [id=91a585d91b924838996b49fe349985f0]
I20251023 20:15:23.619194 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_61af7ade632c46a3becb1573b6313f0c [id=92ce6ecbc3fd4b48af6db9e587adcac5]
I20251023 20:15:23.620556 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_10bf526d2893408887984427efbaa1f2 [id=933b756e69bf4d6d96590acd69f45267]
I20251023 20:15:23.621711 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_515e6170d0e34340a74759ab0a056f72 [id=95f5faaee2b947fea16c52c55657db4a]
I20251023 20:15:23.622067 24184 raft_consensus.cc:493] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.622223 24184 raft_consensus.cc:515] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.622889 24184 leader_election.cc:290] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.624581 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ea656dd8eee44e5f89ad208c70cddc39 [id=96188ab2961f4b14b71df728d91aa4e3]
I20251023 20:15:23.625747 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_f232c9e65a8248d2b818d81486618f5f [id=9834a6ff7acf4269aebcae4712fc7149]
I20251023 20:15:23.625973 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.626295 23670 raft_consensus.cc:2393] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:23.627491 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]
I20251023 20:15:23.629222 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]
I20251023 20:15:23.630851 24230 raft_consensus.cc:515] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.631589 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]
I20251023 20:15:23.632182 24230 leader_election.cc:290] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.632233 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_322349f14bf94a7cbd9cb6cb92793905 [id=9ae6a893fdef4f328cbfd0ee2d57f5a1]
I20251023 20:15:23.634608 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_0febf0b2ad3345998c666d31603db1b1 [id=9dd3c55753fa4ef5af5f31eddb6691b7]
I20251023 20:15:23.634151 24220 raft_consensus.cc:515] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.635860 24220 leader_election.cc:290] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.636507 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_8e0012945b9e401793b99da25bc5d76c [id=a0f3574408d0493e898036800ff93a2f]
I20251023 20:15:23.634768 24186 raft_consensus.cc:2804] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.638792 24186 raft_consensus.cc:697] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.639185 24186 consensus_queue.cc:237] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.639530 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e1d99e08ee354f119f078584381958e6 [id=a34d678a3e1f41d8b65903bc2126e4e8]
I20251023 20:15:23.640949 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_1f76833d42104cee91c1406f649fa028 [id=a48bcc6da6504ccb941c18dd5e289c75]
I20251023 20:15:23.641963 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]
I20251023 20:15:23.642102 23790 tablet_bootstrap.cc:492] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.642294 23790 ts_tablet_manager.cc:1403] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.059s user 0.017s sys 0.008s
I20251023 20:15:23.642779 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b01089ea89ef4536a351098a4caa38f0 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]
I20251023 20:15:23.643234 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48 [id=a8c797d0049e4d1c9123db93f7ea12be]
I20251023 20:15:23.643155 23790 raft_consensus.cc:359] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.643359 23790 raft_consensus.cc:385] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.643442 23790 raft_consensus.cc:740] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.643675 23790 consensus_queue.cc:260] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.644191 23790 ts_tablet_manager.cc:1434] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:23.644310 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4 [id=abf30d62a0a84e239798cc1fca69fea6]
I20251023 20:15:23.644531 23790 tablet_bootstrap.cc:492] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.644901 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_7f5474847aa74113aca18fe96357be62 [id=ac08261334284759a3997df44867cc31]
I20251023 20:15:23.645370 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_20216f4bd8cd4150bfb55aa1e918e7c7 [id=ad2f114d6aaa4bc9a4305c650c90bb23]
I20251023 20:15:23.646100 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]
I20251023 20:15:23.646528 23790 tablet_bootstrap.cc:654] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.646562 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]
I20251023 20:15:23.647473 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_20937abab3a8462cb783c7a4f34a9600 [id=afc43a90036d4b9cb050f975473f2e56]
I20251023 20:15:23.647735 24220 raft_consensus.cc:493] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.647938 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]
I20251023 20:15:23.647966 24220 raft_consensus.cc:515] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.648648 24220 leader_election.cc:290] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.648684 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]
I20251023 20:15:23.649808 23787 tablet_bootstrap.cc:492] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.649904 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]
I20251023 20:15:23.649959 23787 ts_tablet_manager.cc:1403] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.158s user 0.012s sys 0.007s
I20251023 20:15:23.650756 23787 raft_consensus.cc:359] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.650933 23787 raft_consensus.cc:385] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.651010 23787 raft_consensus.cc:740] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.651216 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_316f315bbf2a4332a958a5f72ec606c1 [id=b4bbb2644eaf438f82f912407adc031f]
I20251023 20:15:23.651212 23787 consensus_queue.cc:260] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.651728 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e2c91fae0bf148f198063a62b0426f63 [id=b4c7610c982345c4a6f90e2025b41b2d]
I20251023 20:15:23.652174 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]
W20251023 20:15:23.652231 23333 leader_election.cc:343] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.652712 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]
I20251023 20:15:23.653676 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a6fe726ef844829a7ffec9d4494637" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.653942 23536 raft_consensus.cc:2468] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.654332 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_15836e4048de43f79d9a1d06e0072c3c [id=b9d64b8c30014c5abbb2312d1b8017fe]
I20251023 20:15:23.654703 23787 ts_tablet_manager.cc:1434] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:23.655015 23787 tablet_bootstrap.cc:492] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.655144 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_b589e93e3314462d8ef529985314e501 [id=bdeb476ba31d425893262ab98d5b579f]
I20251023 20:15:23.656097 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dafb9db710f94c328186e3179bf0b319" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.656313 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_57771b43db0747d68d0dea8e558351e0 [id=c35038526af34fc682bc386c5c80cd2d]
I20251023 20:15:23.656311 23533 raft_consensus.cc:3060] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.657212 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]
I20251023 20:15:23.657882 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]
I20251023 20:15:23.658470 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_aba7529cde2c47e39145a97c70f643ed [id=c949af1c17974ec991f6bb58c01c4f2b]
I20251023 20:15:23.658710 23787 tablet_bootstrap.cc:654] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.659008 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]
I20251023 20:15:23.659684 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_48c1e728bef74781a6d7353e1b1181be [id=cd774355ec3e41c29e00778e6fd3c23f]
I20251023 20:15:23.660324 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_8c22652456c84553954550e1018e83e5 [id=ce7dc9691f2a4eeca898b4bf501fae49]
I20251023 20:15:23.661120 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_527cb9b18f414f43872b4f9dd4c433e1 [id=cf913261a84244e0a0ef54347a3a548c]
I20251023 20:15:23.662245 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 [id=d236ba1e610441a8886230b0365ee300]
I20251023 20:15:23.662712 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ce123b77e83b41c4bf1fc149512c47d7 [id=d3624aa88af54933bf909725fd573d76]
I20251023 20:15:23.663173 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_c37d72052930485b819533573f3710ac [id=d38b70b1f8e147e2aeadf0fce048c2a1]
W20251023 20:15:23.663964 23949 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.127.189:33509 timed out after 0.604s (SENT)
I20251023 20:15:23.664319 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_f706cbf159424d419faa383ad30ebf77 [id=d399717713214f3dbc8839f2908ba34a]
I20251023 20:15:23.665170 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]
I20251023 20:15:23.665652 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_43dc7ae468fd42cb867849c762736c66 [id=d53659f9a30541ae913e26824a9ae7d4]
I20251023 20:15:23.666123 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]
I20251023 20:15:23.666258 23533 raft_consensus.cc:2468] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.667135 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.667402 23670 raft_consensus.cc:3060] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.667265 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]
I20251023 20:15:23.668102 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33b5efb30b5c491c8eafe3ac25688843" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.668315 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e72922aae98430da968874f44603317" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.669134 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]
W20251023 20:15:23.669166 23333 leader_election.cc:343] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: f1a6fe726ef844829a7ffec9d4494637
W20251023 20:15:23.669394 23333 leader_election.cc:343] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319
I20251023 20:15:23.669751 23471 leader_election.cc:304] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.669792 24220 raft_consensus.cc:493] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.669921 24220 raft_consensus.cc:515] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.670183 24184 raft_consensus.cc:3060] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.670449 23670 raft_consensus.cc:2468] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.670604 24220 leader_election.cc:290] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:23.670884 24220 raft_consensus.cc:493] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.670822 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33b5efb30b5c491c8eafe3ac25688843" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.671034 23669 raft_consensus.cc:2468] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.671011 24220 raft_consensus.cc:515] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.671785 24220 leader_election.cc:290] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:23.672077 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.672530 23333 leader_election.cc:304] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.672775 24230 raft_consensus.cc:493] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.672891 24230 raft_consensus.cc:515] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.673435 24230 leader_election.cc:290] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.673651 24230 raft_consensus.cc:2804] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.673641 24259 raft_consensus.cc:493] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.673799 24230 raft_consensus.cc:493] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.673800 24259 raft_consensus.cc:515] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.673944 24230 raft_consensus.cc:3060] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.674387 24259 leader_election.cc:290] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.673054 24184 raft_consensus.cc:2749] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:23.675395 23470 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3894dd4feeea4c16861a6c9520059474
I20251023 20:15:23.675688 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]
I20251023 20:15:23.676275 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]
I20251023 20:15:23.677009 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d0548a52907a4b1984665988fa00b425 [id=de89b68b2c194fada173999f7a0af035]
I20251023 20:15:23.677350 23787 tablet_bootstrap.cc:492] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.677529 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]
I20251023 20:15:23.678275 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_6b180ae1e48347588da29cf3bd272d3e [id=e145b28bbae940b0bed6cf3cabd00611]
I20251023 20:15:23.678290 24230 raft_consensus.cc:515] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.678791 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]
I20251023 20:15:23.678799 23787 ts_tablet_manager.cc:1403] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20251023 20:15:23.678925 24230 leader_election.cc:290] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.679096 24230 raft_consensus.cc:493] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.679239 24230 raft_consensus.cc:515] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.679482 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]
I20251023 20:15:23.679622 23787 raft_consensus.cc:359] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.679812 24230 leader_election.cc:290] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.679955 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_7a801afdfd4b4207b0e87c57ee93e01b [id=e74214affb4040149bd7c4db38c87270]
I20251023 20:15:23.680260 23333 leader_election.cc:304] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.680701 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]
W20251023 20:15:23.680797 23333 leader_election.cc:343] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.681245 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_e6177be8581a457cb9113fcc941d6c65 [id=edda395d9ca84731a2bdc438a41aca1a]
I20251023 20:15:23.681254 24220 raft_consensus.cc:2804] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.681455 24220 raft_consensus.cc:697] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.681830 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971]
I20251023 20:15:23.682061 24220 consensus_queue.cc:237] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.682149 23787 raft_consensus.cc:385] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.682237 23787 raft_consensus.cc:740] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.682364 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_69f62f4882e04aa1a32092b258c6acaa [id=ee35b2f67f5f4d9488b1c6484ca4343a]
I20251023 20:15:23.682917 23787 consensus_queue.cc:260] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.683619 24297 raft_consensus.cc:493] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.683713 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]
I20251023 20:15:23.683817 24297 raft_consensus.cc:515] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.684229 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_49f2383c82e44457827030102d18d5bb [id=f49d21d543f5498580620fda6cfc97e9]
I20251023 20:15:23.685067 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_7ab2fc1342c440f9a0c55b4d52c748f8 [id=f59bbf36d431443a85154ff155cb0e66]
I20251023 20:15:23.685745 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_14cd5c04c2b84209a82106f3424a8bd0 [id=f6d53450bf2e46678751ec0bc88efcde]
I20251023 20:15:23.685912 23787 ts_tablet_manager.cc:1434] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251023 20:15:23.686430 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_00e41d94627b471b9753905690b03a49 [id=f9b92376e02047ec9c965526e54739d4]
I20251023 20:15:23.687089 23787 tablet_bootstrap.cc:492] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.687352 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_eb73fed1f9bb425daff2e3278ad80ece [id=fba1e090c19d406c8c9c4728885f1e28]
I20251023 20:15:23.688453 24272 catalog_manager.cc:679] Loaded metadata for table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]
I20251023 20:15:23.691989 23787 tablet_bootstrap.cc:654] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.697187 24297 leader_election.cc:290] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:23.697532 24297 raft_consensus.cc:493] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.697743 24297 raft_consensus.cc:515] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.698310 24297 leader_election.cc:290] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:23.699186 23333 leader_election.cc:343] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.699945 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1229b9e0384a82811f244d2bf4c8ce" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.700470 23533 raft_consensus.cc:2468] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.702040 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0aeac2f20c4ca48c130a0c851f290f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.702055 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0aeac2f20c4ca48c130a0c851f290f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.702246 23669 raft_consensus.cc:2468] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.705289 24272 tablet_loader.cc:96] loaded metadata for tablet 00ec606cbf834ccd8762e034b520e58e (table default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7])
I20251023 20:15:23.705880 24272 tablet_loader.cc:96] loaded metadata for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113])
I20251023 20:15:23.706584 24272 tablet_loader.cc:96] loaded metadata for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_2e38cdb29ca54bdca4c030118988b786 [id=78fe983663c94eabab93ee902bde6998])
I20251023 20:15:23.707734 23787 tablet_bootstrap.cc:492] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.707762 24272 tablet_loader.cc:96] loaded metadata for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a])
I20251023 20:15:23.708559 24272 tablet_loader.cc:96] loaded metadata for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37])
I20251023 20:15:23.709149 23787 ts_tablet_manager.cc:1403] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20251023 20:15:23.709510 24272 tablet_loader.cc:96] loaded metadata for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d (table default.Table_1b20af8a893a4715affda818cffeca13 [id=201b60a519df46fcb92de45acff546cc])
I20251023 20:15:23.710325 24272 tablet_loader.cc:96] loaded metadata for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573])
I20251023 20:15:23.710546 23787 raft_consensus.cc:359] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.711010 24272 tablet_loader.cc:96] loaded metadata for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_31442b53b2f748d0895b0014aa8ae96f [id=90d69f76e2324f5ea1ea4fee2b9042a6])
I20251023 20:15:23.710999 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fbad98d3ab24cc5b7726a92fba04cf8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.711237 23533 raft_consensus.cc:2468] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.711567 24272 tablet_loader.cc:96] loaded metadata for tablet 118be7aa6380469db091d2788472e62b (table default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485])
I20251023 20:15:23.711990 24272 tablet_loader.cc:96] loaded metadata for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6])
I20251023 20:15:23.712697 24272 tablet_loader.cc:96] loaded metadata for tablet 147781e046fe4feb9815147c774fe210 (table default.Table_50879e22af484b719900c9b120534de0 [id=1245153d2d05472a9f20bc69214ca3ce])
I20251023 20:15:23.712719 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5368813164949ff80e27bf59dcc931a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.712930 23787 raft_consensus.cc:385] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.712981 23670 raft_consensus.cc:2468] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.712970 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5368813164949ff80e27bf59dcc931a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.713029 23787 raft_consensus.cc:740] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.713169 23536 raft_consensus.cc:2468] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.713259 23787 consensus_queue.cc:260] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.713339 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.714177 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1a6fe726ef844829a7ffec9d4494637" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.714237 24272 tablet_loader.cc:96] loaded metadata for tablet 165d62dd42da460faccec706f08272a1 (table default.Table_ce123b77e83b41c4bf1fc149512c47d7 [id=d3624aa88af54933bf909725fd573d76])
I20251023 20:15:23.714395 23533 raft_consensus.cc:3060] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.714143 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.714620 23670 raft_consensus.cc:2468] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.715034 24272 tablet_loader.cc:96] loaded metadata for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7])
I20251023 20:15:23.715437 24272 tablet_loader.cc:96] loaded metadata for tablet 19da816a483a4d5b9dc768ef697d54ec (table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641])
I20251023 20:15:23.716336 24272 tablet_loader.cc:96] loaded metadata for tablet 1b220ceff0924b26ae8a54dfb8705456 (table default.Table_b9e2bcb23af94936a9fbfd0817d9a51e [id=3528c0390b7240e8a565945035555dfa])
I20251023 20:15:23.716668 23787 ts_tablet_manager.cc:1434] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:23.717020 24272 tablet_loader.cc:96] loaded metadata for tablet 1e19859a65a144598e1612be89daa3a1 (table default.Table_1ec08f875c4f4a438cd2081db4326eb3 [id=0aefe460f19449a08ef7574681b2af94])
I20251023 20:15:23.717461 24272 tablet_loader.cc:96] loaded metadata for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2 (table default.Table_322349f14bf94a7cbd9cb6cb92793905 [id=9ae6a893fdef4f328cbfd0ee2d57f5a1])
I20251023 20:15:23.717890 24272 tablet_loader.cc:96] loaded metadata for tablet 1e5720c96730425dafb550094cf936cd (table default.Table_37c825c402604e08a862951e693de880 [id=6fdeeacf0d0949838d6062e9330e4524])
I20251023 20:15:23.717928 23787 tablet_bootstrap.cc:492] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.718590 24272 tablet_loader.cc:96] loaded metadata for tablet 1edaed8c82da4e36a312477b15017460 (table default.Table_00e41d94627b471b9753905690b03a49 [id=f9b92376e02047ec9c965526e54739d4])
I20251023 20:15:23.719045 24272 tablet_loader.cc:96] loaded metadata for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973])
I20251023 20:15:23.719934 24272 tablet_loader.cc:96] loaded metadata for tablet 21a508e875c449c5831968562263f1bf (table default.Table_365837ad05624d75855461852362db1c [id=18a582219c6248adac2c8e7e2611808d])
I20251023 20:15:23.720366 24272 tablet_loader.cc:96] loaded metadata for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac])
I20251023 20:15:23.721001 24272 tablet_loader.cc:96] loaded metadata for tablet 225ce6aa52c54f76bd56247f01219dbc (table default.Table_7824bcf5ff034be78ed7dee87e4cd78d [id=44239d72fdcd48258921b9490c822137])
I20251023 20:15:23.721450 24272 tablet_loader.cc:96] loaded metadata for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b (table default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b [id=58cb342d7d6142858153064d66e3712c])
I20251023 20:15:23.722075 24272 tablet_loader.cc:96] loaded metadata for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8])
I20251023 20:15:23.722427 23787 tablet_bootstrap.cc:654] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.723654 24272 tablet_loader.cc:96] loaded metadata for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_1f76833d42104cee91c1406f649fa028 [id=a48bcc6da6504ccb941c18dd5e289c75])
I20251023 20:15:23.724321 24272 tablet_loader.cc:96] loaded metadata for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc])
I20251023 20:15:23.724762 24272 tablet_loader.cc:96] loaded metadata for tablet 27b593b020b34a969f77d2f68178e89d (table default.Table_baf8abd9a132412a8b53ac8d9e044dcf [id=51ea9f032fdd40bb9f0ea42bb80c5cbf])
I20251023 20:15:23.725464 24272 tablet_loader.cc:96] loaded metadata for tablet 29c2c976d6a3440394f478a31abc7134 (table default.Table_43dc7ae468fd42cb867849c762736c66 [id=d53659f9a30541ae913e26824a9ae7d4])
I20251023 20:15:23.725886 24272 tablet_loader.cc:96] loaded metadata for tablet 2a1bf4f01fbc4645bcf8392f56f853b2 (table default.Table_6de6ff0130d74ad894cfa487113e1b19 [id=7a4098cc9d4e49a39ef792883f85d564])
I20251023 20:15:23.726305 24272 tablet_loader.cc:96] loaded metadata for tablet 2c40164e687545c5b2d2751baaee4571 (table default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 [id=6defcb8665844b1d958c71d4fc34263a])
I20251023 20:15:23.727458 24272 tablet_loader.cc:96] loaded metadata for tablet 2e735945647148478201eeffa1d03632 (table default.Table_aba7529cde2c47e39145a97c70f643ed [id=c949af1c17974ec991f6bb58c01c4f2b])
I20251023 20:15:23.728109 24272 tablet_loader.cc:96] loaded metadata for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6])
I20251023 20:15:23.728542 24272 tablet_loader.cc:96] loaded metadata for tablet 31df5c2c40de4d88b6ab1567b84ce4ef (table default.Table_20937abab3a8462cb783c7a4f34a9600 [id=afc43a90036d4b9cb050f975473f2e56])
I20251023 20:15:23.728749 23533 raft_consensus.cc:2468] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:23.729319 23949 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.189:33509, user_credentials={real_user=slave}} blocked reactor thread for 65391.2us
I20251023 20:15:23.729858 24272 tablet_loader.cc:96] loaded metadata for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3])
I20251023 20:15:23.730659 23787 tablet_bootstrap.cc:492] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.730834 23787 ts_tablet_manager.cc:1403] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251023 20:15:23.731385 24272 tablet_loader.cc:96] loaded metadata for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971])
I20251023 20:15:23.731669 23787 raft_consensus.cc:359] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.732147 24272 tablet_loader.cc:96] loaded metadata for tablet 37c8403b53f04a8e97ffe058e59b425c (table default.Table_191b94017cca4cfb946f38ab19ebfe20 [id=330c829b79fd4cce881fc565a3463863])
I20251023 20:15:23.732437 23333 leader_election.cc:304] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.732620 24272 tablet_loader.cc:96] loaded metadata for tablet 385900c60d584c15b702f186ecfbc036 (table default.Table_49f2383c82e44457827030102d18d5bb [id=f49d21d543f5498580620fda6cfc97e9])
I20251023 20:15:23.732964 23333 leader_election.cc:304] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.733351 24272 tablet_loader.cc:96] loaded metadata for tablet 38ce80a338d2494aac84fe6344d47e90 (table default.Table_5e48fffc45314e45a6f7ac8bba5ab31d [id=7260df2ce8034b73956131dd07502d7b])
I20251023 20:15:23.733353 23787 raft_consensus.cc:385] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.733338 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ea63540149c4c02abb8ae51209060a3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.733435 23790 tablet_bootstrap.cc:492] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.733438 23787 raft_consensus.cc:740] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.733564 23790 ts_tablet_manager.cc:1403] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.089s user 0.006s sys 0.014s
I20251023 20:15:23.733721 23787 consensus_queue.cc:260] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.733966 24272 tablet_loader.cc:96] loaded metadata for tablet 39a62c97e0a3459e91c24c1c2f92389a (table default.Table_14cd5c04c2b84209a82106f3424a8bd0 [id=f6d53450bf2e46678751ec0bc88efcde])
I20251023 20:15:23.734030 24230 raft_consensus.cc:2804] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.734191 24230 raft_consensus.cc:697] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.734457 24230 consensus_queue.cc:237] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.734774 24272 tablet_loader.cc:96] loaded metadata for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13])
I20251023 20:15:23.735180 24272 tablet_loader.cc:96] loaded metadata for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_e6177be8581a457cb9113fcc941d6c65 [id=edda395d9ca84731a2bdc438a41aca1a])
I20251023 20:15:23.735580 24272 tablet_loader.cc:96] loaded metadata for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8 (table default.Table_10bf526d2893408887984427efbaa1f2 [id=933b756e69bf4d6d96590acd69f45267])
I20251023 20:15:23.736047 24272 tablet_loader.cc:96] loaded metadata for tablet 3befdfc85bca42bb816e11d1dd832c8c (table default.Table_2496fe6d77e44734bfced35dee7d6596 [id=24a5b1c93ab84b2984243fc1a7f41e0c])
I20251023 20:15:23.736471 24272 tablet_loader.cc:96] loaded metadata for tablet 3e7909dba52748bdb78e08fa4b8a2d47 (table default.Table_e2c91fae0bf148f198063a62b0426f63 [id=b4c7610c982345c4a6f90e2025b41b2d])
I20251023 20:15:23.736472 23787 ts_tablet_manager.cc:1434] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251023 20:15:23.736802 23787 tablet_bootstrap.cc:492] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.738456 24272 tablet_loader.cc:96] loaded metadata for tablet 453eea5cdd2e47f29577c61b40ee59c0 (table default.Table_8c22652456c84553954550e1018e83e5 [id=ce7dc9691f2a4eeca898b4bf501fae49])
I20251023 20:15:23.739154 24272 tablet_loader.cc:96] loaded metadata for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d])
I20251023 20:15:23.740434 24272 tablet_loader.cc:96] loaded metadata for tablet 4bce63866d524067b87145b07347ffcf (table default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8])
I20251023 20:15:23.741112 24272 tablet_loader.cc:96] loaded metadata for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8])
I20251023 20:15:23.741879 24272 tablet_loader.cc:96] loaded metadata for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad])
I20251023 20:15:23.742224 23787 tablet_bootstrap.cc:654] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.743150 24272 tablet_loader.cc:96] loaded metadata for tablet 5161336598884742ad403fc88c9d9f0b (table default.Table_e1d99e08ee354f119f078584381958e6 [id=a34d678a3e1f41d8b65903bc2126e4e8])
I20251023 20:15:23.744521 24272 tablet_loader.cc:96] loaded metadata for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718])
I20251023 20:15:23.745064 24272 tablet_loader.cc:96] loaded metadata for tablet 5407a850ec7045158f53cb8abf6fcb58 (table default.Table_61af7ade632c46a3becb1573b6313f0c [id=92ce6ecbc3fd4b48af6db9e587adcac5])
I20251023 20:15:23.745739 24272 tablet_loader.cc:96] loaded metadata for tablet 559a70cf17ad4145acdb3033e01ad217 (table default.Table_b98d2ec04db54e72a067731a38d21a56 [id=91a585d91b924838996b49fe349985f0])
I20251023 20:15:23.746665 24272 tablet_loader.cc:96] loaded metadata for tablet 58859629a2f14aacbdb5e2cd0edc27bc (table default.Table_f706cbf159424d419faa383ad30ebf77 [id=d399717713214f3dbc8839f2908ba34a])
I20251023 20:15:23.747148 24272 tablet_loader.cc:96] loaded metadata for tablet 58ae6a6e58274b8baf952eaaefde7127 (table default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f])
I20251023 20:15:23.747653 24272 tablet_loader.cc:96] loaded metadata for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_0febf0b2ad3345998c666d31603db1b1 [id=9dd3c55753fa4ef5af5f31eddb6691b7])
I20251023 20:15:23.748256 24186 raft_consensus.cc:2804] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.748418 24186 raft_consensus.cc:493] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.748447 24272 tablet_loader.cc:96] loaded metadata for tablet 5acdc66df9c748a1a6300962f62fcf36 (table default.Table_20216f4bd8cd4150bfb55aa1e918e7c7 [id=ad2f114d6aaa4bc9a4305c650c90bb23])
I20251023 20:15:23.748615 24186 raft_consensus.cc:3060] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.748643 23790 raft_consensus.cc:359] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.748855 23790 raft_consensus.cc:385] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.748950 23790 raft_consensus.cc:740] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.748981 24272 tablet_loader.cc:96] loaded metadata for tablet 5cdde6b251b14df4bf44142ce263b84a (table default.Table_30b62bd8bf6147a7a5800584576db269 [id=054cce203d7e4437a8a7aa22df487453])
I20251023 20:15:23.749207 23790 consensus_queue.cc:260] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.749511 24272 tablet_loader.cc:96] loaded metadata for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_4fb0e33f75594ca59d2ecb391873d804 [id=1a32884612a34df48b257e1a18bb5e73])
I20251023 20:15:23.749799 23790 ts_tablet_manager.cc:1434] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251023 20:15:23.750174 23790 tablet_bootstrap.cc:492] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.750698 24272 tablet_loader.cc:96] loaded metadata for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477])
I20251023 20:15:23.751521 24272 tablet_loader.cc:96] loaded metadata for tablet 60e8942deb7544a09b21c3d1c2a3f99f (table default.Table_d0548a52907a4b1984665988fa00b425 [id=de89b68b2c194fada173999f7a0af035])
I20251023 20:15:23.751839 24186 raft_consensus.cc:515] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.752128 24272 tablet_loader.cc:96] loaded metadata for tablet 6313d3307aff4a6880176a06627e86fc (table default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 [id=3973b56caa79477f8542118421741084])
I20251023 20:15:23.752518 24272 tablet_loader.cc:96] loaded metadata for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066])
I20251023 20:15:23.752878 24272 tablet_loader.cc:96] loaded metadata for tablet 63ef1bae87c54aa988dfe49f828fe4f8 (table default.Table_6443f81020254f45a1b7fd93ad217e09 [id=4d43a0eda13041689a414ecbadb85e09])
I20251023 20:15:23.753945 24272 tablet_loader.cc:96] loaded metadata for tablet 660d84eb63a4490886aba1777d449ee2 (table default.Table_ec09021e59f8464f81cb6ceda1f1c567 [id=2fd818d424a44efdab6cf6fda7537ee8])
I20251023 20:15:23.754710 23787 tablet_bootstrap.cc:492] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.755368 24272 tablet_loader.cc:96] loaded metadata for tablet 67e0701159c548c49e9e245820c45028 (table default.Table_7f5474847aa74113aca18fe96357be62 [id=ac08261334284759a3997df44867cc31])
I20251023 20:15:23.755597 23787 ts_tablet_manager.cc:1403] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20251023 20:15:23.756402 23787 raft_consensus.cc:359] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.757154 24272 tablet_loader.cc:96] loaded metadata for tablet 6abfda15e6c242b588d4ed4401008156 (table default.Table_15836e4048de43f79d9a1d06e0072c3c [id=b9d64b8c30014c5abbb2312d1b8017fe])
I20251023 20:15:23.757580 23787 raft_consensus.cc:385] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.757725 23787 raft_consensus.cc:740] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.758283 23787 consensus_queue.cc:260] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.758827 24272 tablet_loader.cc:96] loaded metadata for tablet 6e4189623c7a4ca1bdf73f71663e2b77 (table default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 [id=d236ba1e610441a8886230b0365ee300])
I20251023 20:15:23.759316 23787 ts_tablet_manager.cc:1434] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:23.759720 24272 tablet_loader.cc:96] loaded metadata for tablet 6ee1974eb5e343b7953dadda7d5799b1 (table default.Table_515e6170d0e34340a74759ab0a056f72 [id=95f5faaee2b947fea16c52c55657db4a])
I20251023 20:15:23.760596 24272 tablet_loader.cc:96] loaded metadata for tablet 72d1dde8b0ac4812869ef233c18e550f (table default.Table_efda7d26e2124ec4b1eeca32fe1b545e [id=6f9ad1c9f64a44a4b856013bb8fd4bd1])
I20251023 20:15:23.760921 23787 tablet_bootstrap.cc:492] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.763278 23787 tablet_bootstrap.cc:654] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.764520 24272 tablet_loader.cc:96] loaded metadata for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e])
I20251023 20:15:23.765167 24272 tablet_loader.cc:96] loaded metadata for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912])
I20251023 20:15:23.766503 24272 tablet_loader.cc:96] loaded metadata for tablet 76380b341a304d6ba31648fbd1548f05 (table default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277])
I20251023 20:15:23.768251 24272 tablet_loader.cc:96] loaded metadata for tablet 7f2a4f1d52984af58d6286671389b6a9 (table default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525])
I20251023 20:15:23.769426 24272 tablet_loader.cc:96] loaded metadata for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc])
I20251023 20:15:23.771047 24272 tablet_loader.cc:96] loaded metadata for tablet 882f7bf08e2f4b5382702c3d6eaf201e (table default.Table_39fdd0d19cae4e928fdee07e74900b4b [id=356410224c0f44bdacbaa543e9dee27d])
I20251023 20:15:23.772238 24272 tablet_loader.cc:96] loaded metadata for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae])
I20251023 20:15:23.773039 24272 tablet_loader.cc:96] loaded metadata for tablet 8a45d9b3031b46d390e0a1010489c718 (table default.Table_ea656dd8eee44e5f89ad208c70cddc39 [id=96188ab2961f4b14b71df728d91aa4e3])
I20251023 20:15:23.773906 24272 tablet_loader.cc:96] loaded metadata for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249])
I20251023 20:15:23.774768 24272 tablet_loader.cc:96] loaded metadata for tablet 8d22aa40e5844d78a8679a3ad1b08da9 (table default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 [id=79163b61e42a43d8879399cd5f31ce44])
I20251023 20:15:23.775080 23533 raft_consensus.cc:2468] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.752269 23790 tablet_bootstrap.cc:654] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.775875 24272 tablet_loader.cc:96] loaded metadata for tablet 8d9b192c111c47d1983e9a48d07da288 (table default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d [id=5a207d264756433f94af5164462692ff])
I20251023 20:15:23.777017 24272 tablet_loader.cc:96] loaded metadata for tablet 8e718cb7205643fd8c3045c3fab30d3e (table default.Table_464dc629d50b4995b30ef5c463e9f316 [id=8477c10fb2f4422996d4eab8717d7fe8])
I20251023 20:15:23.778353 24272 tablet_loader.cc:96] loaded metadata for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c])
I20251023 20:15:23.779698 24272 tablet_loader.cc:96] loaded metadata for tablet 92576756ec574ceaa4c81f435cb72301 (table default.Table_eb73fed1f9bb425daff2e3278ad80ece [id=fba1e090c19d406c8c9c4728885f1e28])
I20251023 20:15:23.780959 24272 tablet_loader.cc:96] loaded metadata for tablet 92c50c4845524163834861438291a3ec (table default.Table_b01089ea89ef4536a351098a4caa38f0 [id=a8bd87490f4d46b7ad7df86a5ad6e11e])
I20251023 20:15:23.782444 24272 tablet_loader.cc:96] loaded metadata for tablet 94cd8880a78c4b4eb0c19ba5944b12f2 (table default.Table_240b16ca6e1543f2aab607c1d6a447fd [id=4aa2d573bbb34a3ebda2ac6376a387ea])
I20251023 20:15:23.782661 24184 raft_consensus.cc:493] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.782859 24184 raft_consensus.cc:515] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.783010 24186 leader_election.cc:290] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.783558 24184 leader_election.cc:290] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:23.784356 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.784704 23670 raft_consensus.cc:2393] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:23.784886 24272 tablet_loader.cc:96] loaded metadata for tablet 95a06da281194f95b126483314f6debd (table default.Table_69f62f4882e04aa1a32092b258c6acaa [id=ee35b2f67f5f4d9488b1c6484ca4343a])
I20251023 20:15:23.785346 23786 tablet_bootstrap.cc:492] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.785516 23786 ts_tablet_manager.cc:1403] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.183s user 0.054s sys 0.051s
I20251023 20:15:23.785777 24272 tablet_loader.cc:96] loaded metadata for tablet 9749c6f3fe5545ab93bf49aae97e9818 (table default.Table_b589e93e3314462d8ef529985314e501 [id=bdeb476ba31d425893262ab98d5b579f])
I20251023 20:15:23.786127 24272 tablet_loader.cc:96] loaded metadata for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b])
I20251023 20:15:23.786281 23786 raft_consensus.cc:359] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:23.786381 23038 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu pid 23114: Timed out: CheckLeaks RPC to 127.22.127.189:33509 timed out after 1.000s (SENT)
I20251023 20:15:23.786478 23786 raft_consensus.cc:385] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.786554 23786 raft_consensus.cc:740] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.786578 23038 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu with pid 23114
I20251023 20:15:23.787076 23786 consensus_queue.cc:260] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.787575 24272 tablet_loader.cc:96] loaded metadata for tablet 98a4553c1f014ef3ac975bf7a8076334 (table default.Table_ac4b704784f2423a82ad591bbed856ab [id=3cf9ba4080ce4e5ead110ed71e43ce1f])
I20251023 20:15:23.787807 24184 raft_consensus.cc:493] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.787954 24184 raft_consensus.cc:515] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.788201 24272 tablet_loader.cc:96] loaded metadata for tablet 98a6a3425a49403e99c8ade114166e46 (table default.Table_316f315bbf2a4332a958a5f72ec606c1 [id=b4bbb2644eaf438f82f912407adc031f])
I20251023 20:15:23.788547 24272 tablet_loader.cc:96] loaded metadata for tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca])
I20251023 20:15:23.788576 24184 leader_election.cc:290] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:23.774921 23333 leader_election.cc:343] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ef1229b9e0384a82811f244d2bf4c8ce
I20251023 20:15:23.789134 24272 tablet_loader.cc:96] loaded metadata for tablet 9a4f012469fa4e8e827e485287ad76dd (table default.Table_e6e2be420a6342bc9d1311bd4fe6080c [id=8eeb69b1cebc42dcaae9f3306aa18013])
I20251023 20:15:23.789172 23786 ts_tablet_manager.cc:1434] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:23.789425 23333 leader_election.cc:304] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.789502 23786 tablet_bootstrap.cc:492] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.789701 24272 tablet_loader.cc:96] loaded metadata for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664])
W20251023 20:15:23.790361 23333 leader_election.cc:343] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3fbad98d3ab24cc5b7726a92fba04cf8
I20251023 20:15:23.790709 23787 tablet_bootstrap.cc:492] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.790915 23787 ts_tablet_manager.cc:1403] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.006s
I20251023 20:15:23.791435 23786 tablet_bootstrap.cc:654] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.791711 23787 raft_consensus.cc:359] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.791967 23787 raft_consensus.cc:385] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.792084 23787 raft_consensus.cc:740] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.792392 23787 consensus_queue.cc:260] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.792891 23787 ts_tablet_manager.cc:1434] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.793287 23787 tablet_bootstrap.cc:492] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.794920 23787 tablet_bootstrap.cc:654] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:23.795265 23333 leader_election.cc:343] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: f1a6fe726ef844829a7ffec9d4494637
I20251023 20:15:23.795776 23333 leader_election.cc:304] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.796056 24272 tablet_loader.cc:96] loaded metadata for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6])
I20251023 20:15:23.796411 24230 raft_consensus.cc:2804] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.796531 24230 raft_consensus.cc:493] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.796653 24230 raft_consensus.cc:3060] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.796680 24272 tablet_loader.cc:96] loaded metadata for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d])
I20251023 20:15:23.797508 24272 tablet_loader.cc:96] loaded metadata for tablet a0079496170849c6b3c233e1cd78e40e (table default.Table_3a5e9b90e86340ed80fdb4a8679777e6 [id=22d88af65ebd43389f6e58573143816c])
I20251023 20:15:23.797864 24272 tablet_loader.cc:96] loaded metadata for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b])
I20251023 20:15:23.798211 24272 tablet_loader.cc:96] loaded metadata for tablet a1efa4fcaeb049b484ca59b12025ce9a (table default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4 [id=abf30d62a0a84e239798cc1fca69fea6])
I20251023 20:15:23.799224 24272 tablet_loader.cc:96] loaded metadata for tablet a2c022d78cac4959984d608d64df851b (table default.Table_698f8853c4774ccdae2e37d28f90f368 [id=3e597b14c5d944838c32914dc1779d20])
I20251023 20:15:23.799609 24272 tablet_loader.cc:96] loaded metadata for tablet a2d4ba34cc444baf9fe725bc81478517 (table default.Table_87caa0c69120413a92c6b65337baad14 [id=3227f5fd7d78482c99fb9f654ba8e3c6])
I20251023 20:15:23.799659 24230 raft_consensus.cc:515] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.800309 24272 tablet_loader.cc:96] loaded metadata for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01])
I20251023 20:15:23.800488 24230 leader_election.cc:290] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:23.800710 24230 raft_consensus.cc:2804] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.800835 24230 raft_consensus.cc:493] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.800944 23786 tablet_bootstrap.cc:492] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.800943 24230 raft_consensus.cc:3060] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.801134 23786 ts_tablet_manager.cc:1403] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20251023 20:15:23.801486 24272 tablet_loader.cc:96] loaded metadata for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e])
I20251023 20:15:23.801962 23786 raft_consensus.cc:359] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.802131 23786 raft_consensus.cc:385] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.802227 23786 raft_consensus.cc:740] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.802302 24272 tablet_loader.cc:96] loaded metadata for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f])
W20251023 20:15:23.802448 23470 leader_election.cc:343] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 23225d31c2e647398aa58494c627507e
I20251023 20:15:23.802798 23786 consensus_queue.cc:260] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.803409 23786 ts_tablet_manager.cc:1434] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.803715 23786 tablet_bootstrap.cc:492] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.803889 24272 tablet_loader.cc:96] loaded metadata for tablet a7f9428277fb4ebf989fed7569dee687 (table default.Table_9cac41d47a584e59a3c3e6fd69f9ad79 [id=6b26d29083ab46a29f00f2552ef20892])
I20251023 20:15:23.804011 24230 raft_consensus.cc:515] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.804627 24230 leader_election.cc:290] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.804912 23333 leader_election.cc:304] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:23.805203 23333 leader_election.cc:343] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 4ea63540149c4c02abb8ae51209060a3
I20251023 20:15:23.805342 24272 tablet_loader.cc:96] loaded metadata for tablet a9e5dc42c01a4d5e9661f399f793f68f (table default.Table_e851810007344c51aa7a54934d6ad579 [id=0ddefd3e67364bc8bf23971811893894])
I20251023 20:15:23.805728 24272 tablet_loader.cc:96] loaded metadata for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6])
I20251023 20:15:23.805905 23333 leader_election.cc:304] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:23.806288 23333 leader_election.cc:343] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.806486 24272 tablet_loader.cc:96] loaded metadata for tablet ac4d82d024fc491ab0dbe04147cb156c (table default.Table_baf578c1b5e748e09f244f10685992a8 [id=006294844eac4bcabdd6508738bdc4cf])
I20251023 20:15:23.806612 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33b5efb30b5c491c8eafe3ac25688843" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.806818 23670 raft_consensus.cc:3060] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:23.807286 23471 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 3894dd4feeea4c16861a6c9520059474
I20251023 20:15:23.807410 23471 leader_election.cc:304] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.807603 23786 tablet_bootstrap.cc:654] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.808123 24184 raft_consensus.cc:2749] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.808182 23787 tablet_bootstrap.cc:492] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.808328 23787 ts_tablet_manager.cc:1403] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20251023 20:15:23.809217 23787 raft_consensus.cc:359] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.809284 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0aeac2f20c4ca48c130a0c851f290f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.809383 23787 raft_consensus.cc:385] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.809474 23669 raft_consensus.cc:3060] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.809477 23787 raft_consensus.cc:740] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.809752 23787 consensus_queue.cc:260] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.810303 23787 ts_tablet_manager.cc:1434] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.810596 23787 tablet_bootstrap.cc:492] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.811080 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0aeac2f20c4ca48c130a0c851f290f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.811525 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5368813164949ff80e27bf59dcc931a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.811678 23536 raft_consensus.cc:3060] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.812305 23787 tablet_bootstrap.cc:654] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.812700 23669 raft_consensus.cc:2468] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.813112 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5368813164949ff80e27bf59dcc931a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.813337 23668 raft_consensus.cc:3060] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.813998 24272 tablet_loader.cc:96] loaded metadata for tablet aec7c5ea622949aca6c3dbe0e5f8e81e (table default.Table_4c838e5585b948b9ba710086a76a2f3f [id=04d988798f7f4b56be2fe5ed680539d4])
I20251023 20:15:23.814364 24272 tablet_loader.cc:96] loaded metadata for tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399])
I20251023 20:15:23.815176 24272 tablet_loader.cc:96] loaded metadata for tablet b30e8ca2c96f4063a7f0e8f324c78b12 (table default.Table_6b180ae1e48347588da29cf3bd272d3e [id=e145b28bbae940b0bed6cf3cabd00611])
I20251023 20:15:23.815491 24272 tablet_loader.cc:96] loaded metadata for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_4f8238f0492b4364b731c0bf5e2bd53a [id=2db1100447a44e7fb43e87cdf7247d14])
I20251023 20:15:23.815806 23786 tablet_bootstrap.cc:492] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.815954 23786 ts_tablet_manager.cc:1403] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20251023 20:15:23.816011 23668 raft_consensus.cc:2468] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.816687 23786 raft_consensus.cc:359] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.816890 23786 raft_consensus.cc:385] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.816968 23786 raft_consensus.cc:740] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.817229 23786 consensus_queue.cc:260] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.817323 23670 raft_consensus.cc:2468] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.817477 24272 tablet_loader.cc:96] loaded metadata for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346])
I20251023 20:15:23.817780 23786 ts_tablet_manager.cc:1434] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:23.817811 23536 raft_consensus.cc:2468] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.818295 23786 tablet_bootstrap.cc:492] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.819300 24272 tablet_loader.cc:96] loaded metadata for tablet b446550686024c3ca80f106112a2110f (table default.Table_7a801afdfd4b4207b0e87c57ee93e01b [id=e74214affb4040149bd7c4db38c87270])
I20251023 20:15:23.819676 24272 tablet_loader.cc:96] loaded metadata for tablet b4798f6eea5745ba8a0d4b8cb173cc5c (table default.Table_8dc2f44cc1fc43e4a9851a0587a74edb [id=3bf5870046244dcd817441dce750a1e2])
I20251023 20:15:23.819856 23786 tablet_bootstrap.cc:654] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.820011 24272 tablet_loader.cc:96] loaded metadata for tablet b5cb6b4e2e5e4a2f9183111cfd54ec90 (table default.Table_57771b43db0747d68d0dea8e558351e0 [id=c35038526af34fc682bc386c5c80cd2d])
I20251023 20:15:23.820370 24272 tablet_loader.cc:96] loaded metadata for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f])
I20251023 20:15:23.820699 24272 tablet_loader.cc:96] loaded metadata for tablet b8e41ac34e84459581daef13914942b5 (table default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6 [id=77ff0c7e54e94defabffdbc82dbb609d])
I20251023 20:15:23.821935 24272 tablet_loader.cc:96] loaded metadata for tablet c05b9bcea6ff49188758271401270227 (table default.Table_f232c9e65a8248d2b818d81486618f5f [id=9834a6ff7acf4269aebcae4712fc7149])
I20251023 20:15:23.825418 24272 tablet_loader.cc:96] loaded metadata for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 (table default.Table_c37d72052930485b819533573f3710ac [id=d38b70b1f8e147e2aeadf0fce048c2a1])
I20251023 20:15:23.826208 24272 tablet_loader.cc:96] loaded metadata for tablet c24c33c11d0d4d22bdcb96e1cf88ad66 (table default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063])
I20251023 20:15:23.827096 24272 tablet_loader.cc:96] loaded metadata for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c])
I20251023 20:15:23.828084 24272 tablet_loader.cc:96] loaded metadata for tablet c3d145a33b964a4ca1f0001ac6cf990b (table default.Table_7ab2fc1342c440f9a0c55b4d52c748f8 [id=f59bbf36d431443a85154ff155cb0e66])
I20251023 20:15:23.828886 24272 tablet_loader.cc:96] loaded metadata for tablet c3d238e071cb4d49b173febd90bc909b (table default.Table_d491cd3951344335be8170dcdcb94c4c [id=2c61abbba33e4ed4805127246e3cd0ba])
I20251023 20:15:23.829501 23333 leader_election.cc:304] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.829679 24272 tablet_loader.cc:96] loaded metadata for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55])
I20251023 20:15:23.819026 23790 tablet_bootstrap.cc:492] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.829918 23790 ts_tablet_manager.cc:1403] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.006s
I20251023 20:15:23.830828 23790 raft_consensus.cc:359] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.831048 23790 raft_consensus.cc:385] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.831120 23790 raft_consensus.cc:740] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.831359 23790 consensus_queue.cc:260] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.831920 23790 ts_tablet_manager.cc:1434] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:23.832306 23790 tablet_bootstrap.cc:492] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.834036 24272 tablet_loader.cc:96] loaded metadata for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_48c1e728bef74781a6d7353e1b1181be [id=cd774355ec3e41c29e00778e6fd3c23f])
I20251023 20:15:23.834241 23790 tablet_bootstrap.cc:654] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.817710 24230 raft_consensus.cc:2804] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.835790 24272 tablet_loader.cc:96] loaded metadata for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9])
I20251023 20:15:23.835843 24230 raft_consensus.cc:493] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.836082 24230 raft_consensus.cc:3060] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.837589 24272 tablet_loader.cc:96] loaded metadata for tablet d1cd0a61827c49fd99dfd054c9a292f8 (table default.Table_f5ea72074b174da28b3738378167f49c [id=70e0efabed5a4dfe98eb7201a2b4aba2])
I20251023 20:15:23.838615 24297 raft_consensus.cc:2804] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.838780 24297 raft_consensus.cc:493] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.838960 24297 raft_consensus.cc:3060] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.839179 24272 tablet_loader.cc:96] loaded metadata for tablet d227b45bfcc64d76a51cff347e6e1486 (table default.Table_a5331f0f356d4ed88cf6787eb94db083 [id=7bf105aabdc944afac7c5d1e5afb824b])
I20251023 20:15:23.839314 24230 raft_consensus.cc:515] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.839991 24272 tablet_loader.cc:96] loaded metadata for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48])
I20251023 20:15:23.840831 24272 tablet_loader.cc:96] loaded metadata for tablet d62dbf044acd4459b26d2e7e50f6206e (table default.Table_8e0012945b9e401793b99da25bc5d76c [id=a0f3574408d0493e898036800ff93a2f])
I20251023 20:15:23.841823 24220 raft_consensus.cc:2804] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.842242 24220 raft_consensus.cc:493] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.842340 24220 raft_consensus.cc:3060] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.842705 24230 leader_election.cc:290] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:23.843377 24272 tablet_loader.cc:96] loaded metadata for tablet d6b0a4db18df490a966e8c475ddd14f4 (table default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12 [id=6eda60ed5ff344f3a610b8139aa5a6f5])
I20251023 20:15:23.844087 23790 tablet_bootstrap.cc:492] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.844235 23790 ts_tablet_manager.cc:1403] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20251023 20:15:23.845116 23790 raft_consensus.cc:359] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.845317 23790 raft_consensus.cc:385] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.845415 23790 raft_consensus.cc:740] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.845681 23790 consensus_queue.cc:260] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.846212 23790 ts_tablet_manager.cc:1434] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:23.846246 23786 tablet_bootstrap.cc:492] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.846377 23786 ts_tablet_manager.cc:1403] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.016s
I20251023 20:15:23.846544 23790 tablet_bootstrap.cc:492] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.847136 24272 tablet_loader.cc:96] loaded metadata for tablet dae906f257c346d7a1b28c9052d600b5 (table default.Table_775428d3af964d538226c836574a899d [id=818f0fca06904f09ad46c9e184c982b4])
I20251023 20:15:23.847117 23786 raft_consensus.cc:359] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.847298 23786 raft_consensus.cc:385] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.847384 23786 raft_consensus.cc:740] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.847601 23786 consensus_queue.cc:260] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.848126 23786 ts_tablet_manager.cc:1434] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:23.848229 23790 tablet_bootstrap.cc:654] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.848429 23786 tablet_bootstrap.cc:492] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.849633 24272 tablet_loader.cc:96] loaded metadata for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582])
I20251023 20:15:23.850426 23786 tablet_bootstrap.cc:654] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.850436 24013 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Refusing update from remote peer 690ffde75cd745b6a87925967d54471a: Log matching property violated. Preceding OpId in replica: term: 1 index: 633. Preceding OpId from leader: term: 2 index: 659. (index mismatch)
W20251023 20:15:23.850186 23203 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.127.189:33509 timed out after 0.512s (SENT)
I20251023 20:15:23.851568 24272 tablet_loader.cc:96] loaded metadata for tablet dd07383242b74637bac46609f56c7e4d (table default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b [id=905ffd6fc69a44c39b12a7efecc70fd4])
I20251023 20:15:23.851578 24265 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 659, Last known committed idx: 626, Time since last communication: 0.000s
I20251023 20:15:23.852833 24272 tablet_loader.cc:96] loaded metadata for tablet e18c1df87ee640148f5e827ee168fea6 (table default.Table_c760c475c8554c89aeb6fd77fd422797 [id=886ab40a90ab4410aa13fabf17a1ea91])
I20251023 20:15:23.854593 24272 tablet_loader.cc:96] loaded metadata for tablet e3df1fd83e3c4ac29a7846736701044a (table default.Table_3ad1b2da425540ccbe8b55d22f379b4a [id=7810e4d5193c4559891f1603fe361e11])
I20251023 20:15:23.854900 24174 raft_consensus.cc:493] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.855059 24174 raft_consensus.cc:515] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.855748 24174 leader_election.cc:290] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.856627 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e1b4748c0244b258960b1025da47562" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.856767 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e1b4748c0244b258960b1025da47562" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.856892 24272 tablet_loader.cc:96] loaded metadata for tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9])
W20251023 20:15:23.857314 23604 leader_election.cc:343] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:23.857738 23605 leader_election.cc:343] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.857854 23605 leader_election.cc:304] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.858364 24174 raft_consensus.cc:2749] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.858429 24272 tablet_loader.cc:96] loaded metadata for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7])
I20251023 20:15:23.858795 24272 tablet_loader.cc:96] loaded metadata for tablet e658aa360e524a9f9bdb35c1f68ae426 (table default.Table_25427c3d36fb4ba4a01ac362787c81b8 [id=248b68f25e64412ba27e4c74b27c59d8])
I20251023 20:15:23.860316 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33b5efb30b5c491c8eafe3ac25688843" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.860622 24272 tablet_loader.cc:96] loaded metadata for tablet e9e2fec8f4e14036af714427481f2f36 (table default.Table_de4427f05119413b8983f4e818072f98 [id=591b99acc8dd4e7388acb07fdf88a50a])
I20251023 20:15:23.861243 24272 tablet_loader.cc:96] loaded metadata for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48 [id=a8c797d0049e4d1c9123db93f7ea12be])
I20251023 20:15:23.861263 24220 raft_consensus.cc:515] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.860886 23533 raft_consensus.cc:3060] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.862672 24272 tablet_loader.cc:96] loaded metadata for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815])
I20251023 20:15:23.864586 24272 tablet_loader.cc:96] loaded metadata for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f])
I20251023 20:15:23.864626 23533 raft_consensus.cc:2468] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.866155 24272 tablet_loader.cc:96] loaded metadata for tablet f03df77c344b4736a30725e4d9f72a35 (table default.Table_7908a191ee99400093d9f2deb05e52d2 [id=2ad9425327fb485d89c61e102c983b30])
I20251023 20:15:23.867413 24272 tablet_loader.cc:96] loaded metadata for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3])
I20251023 20:15:23.868685 24272 tablet_loader.cc:96] loaded metadata for tablet f4023a5776334f42a2564f538a7c5fee (table default.Table_527cb9b18f414f43872b4f9dd4c433e1 [id=cf913261a84244e0a0ef54347a3a548c])
I20251023 20:15:23.868994 24220 leader_election.cc:290] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.870440 24272 tablet_loader.cc:96] loaded metadata for tablet fa0a46ffa626447eab4cdb956e798005 (table default.Table_934c5152009843bcbf11132bbe4c8ed2 [id=09f8335a5b03409c90559c8313f5d135])
I20251023 20:15:23.873212 24272 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251023 20:15:23.871243 24297 raft_consensus.cc:515] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.878892 24297 leader_election.cc:290] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:23.842044 23333 leader_election.cc:343] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:23.889230 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 47200.8us
I20251023 20:15:23.873562 23786 tablet_bootstrap.cc:492] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.889909 23786 ts_tablet_manager.cc:1403] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.006s
I20251023 20:15:23.890406 24272 catalog_manager.cc:1269] Loaded cluster ID: 67ec0cebd86e4322972b698542d64fab
I20251023 20:15:23.891387 23333 leader_election.cc:304] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.892305 23786 raft_consensus.cc:359] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.892565 23786 raft_consensus.cc:385] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.892655 23786 raft_consensus.cc:740] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.892951 23786 consensus_queue.cc:260] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.893563 23786 ts_tablet_manager.cc:1434] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:23.894076 23786 tablet_bootstrap.cc:492] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:23.894099 24272 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251023 20:15:23.895824 23786 tablet_bootstrap.cc:654] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.902899 23333 leader_election.cc:304] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.903776 24186 raft_consensus.cc:2804] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.895756 24297 raft_consensus.cc:493] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.904763 24297 raft_consensus.cc:515] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.905478 24297 leader_election.cc:290] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.905882 24220 raft_consensus.cc:2804] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.910203 24184 raft_consensus.cc:493] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.910383 24184 raft_consensus.cc:515] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.911072 24184 leader_election.cc:290] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:23.906008 24220 raft_consensus.cc:493] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.912117 24272 catalog_manager.cc:1514] Loading token signing keys...
I20251023 20:15:23.912119 24220 raft_consensus.cc:3060] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.912060 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0aeac2f20c4ca48c130a0c851f290f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:23.912338 23670 raft_consensus.cc:2393] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:23.915385 24220 raft_consensus.cc:515] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.915728 24186 raft_consensus.cc:697] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.917924 24186 consensus_queue.cc:237] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.919076 24174 raft_consensus.cc:493] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.919293 24174 raft_consensus.cc:515] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.920097 24174 leader_election.cc:290] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.921196 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a27f192792cf41288d69ca15ab538472" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:23.921839 23605 leader_election.cc:343] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.924952 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a27f192792cf41288d69ca15ab538472" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:23.925604 23604 leader_election.cc:343] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.925724 23604 leader_election.cc:304] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.926193 24174 raft_consensus.cc:2749] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:23.916705 23333 leader_election.cc:343] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.928619 24272 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Loaded TSK: 0
I20251023 20:15:23.928849 23333 leader_election.cc:304] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.929492 23333 leader_election.cc:304] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:23.932667 24297 raft_consensus.cc:2804] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.932940 24297 raft_consensus.cc:697] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.933437 24297 consensus_queue.cc:237] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.934492 24272 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251023 20:15:23.929740 24186 raft_consensus.cc:2804] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.934743 24186 raft_consensus.cc:697] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.935063 24186 consensus_queue.cc:237] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.931453 24220 leader_election.cc:290] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:23.946151 23203 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.189:33509, user_credentials={real_user=slave}} blocked reactor thread for 96022us
I20251023 20:15:23.949257 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5eab3e95be0f41dcb5a8ec63e91f5d8e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.956781 23786 tablet_bootstrap.cc:492] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:23.957391 23786 ts_tablet_manager.cc:1403] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.063s user 0.018s sys 0.008s
I20251023 20:15:23.957149 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.958796 23786 raft_consensus.cc:359] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.959074 23786 raft_consensus.cc:385] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.959200 23786 raft_consensus.cc:740] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.959436 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:23.959692 23670 raft_consensus.cc:3060] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.959584 23786 consensus_queue.cc:260] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:23.960821 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef1229b9e0384a82811f244d2bf4c8ce" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.961014 23536 raft_consensus.cc:3060] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.962733 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fbad98d3ab24cc5b7726a92fba04cf8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.962905 23533 raft_consensus.cc:3060] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.962908 24174 raft_consensus.cc:493] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.963054 24174 raft_consensus.cc:515] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.963721 24174 leader_election.cc:290] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.964243 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4beebee06e5341a29c65d0c3cf985ef4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.964541 23670 raft_consensus.cc:2468] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:23.964949 23604 leader_election.cc:343] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.964913 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4beebee06e5341a29c65d0c3cf985ef4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.965420 23399 raft_consensus.cc:2468] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:23.965907 23605 leader_election.cc:304] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.966264 23536 raft_consensus.cc:2468] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.966398 24174 raft_consensus.cc:2804] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:23.966499 24174 raft_consensus.cc:493] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:23.966528 23533 raft_consensus.cc:2468] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.966591 24174 raft_consensus.cc:3060] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.967279 24160 raft_consensus.cc:493] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:23.967417 24160 raft_consensus.cc:515] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.968050 24160 leader_election.cc:290] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.969022 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7301d6121c9d4c8dbc642e895aa4e9f6" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:23.969359 23536 raft_consensus.cc:2468] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:23.969349 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:23.969924 23605 leader_election.cc:343] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.970405 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4ea63540149c4c02abb8ae51209060a3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.970731 23536 raft_consensus.cc:3060] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.971372 24174 raft_consensus.cc:515] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.972399 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4beebee06e5341a29c65d0c3cf985ef4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:23.972592 23399 raft_consensus.cc:3060] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.973364 23536 raft_consensus.cc:2468] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:23.973920 23471 leader_election.cc:304] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.974325 24184 raft_consensus.cc:3060] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.975287 23399 raft_consensus.cc:2468] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:23.975796 23605 leader_election.cc:304] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:23.976303 24174 leader_election.cc:290] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:23.976478 24174 raft_consensus.cc:2804] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.976791 24184 raft_consensus.cc:2749] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.977113 24174 raft_consensus.cc:697] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:23.977381 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4beebee06e5341a29c65d0c3cf985ef4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:23.977402 24174 consensus_queue.cc:237] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:23.977890 23604 leader_election.cc:343] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:23.977936 23471 leader_election.cc:343] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 23225d31c2e647398aa58494c627507e
I20251023 20:15:23.978047 23471 leader_election.cc:304] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.978485 24184 raft_consensus.cc:2749] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.978893 23333 leader_election.cc:304] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:23.985896 23333 leader_election.cc:343] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3fbad98d3ab24cc5b7726a92fba04cf8
I20251023 20:15:23.986178 24297 raft_consensus.cc:2804] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.986613 24297 raft_consensus.cc:697] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:23.986680 23787 tablet_bootstrap.cc:492] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:23.986955 23787 ts_tablet_manager.cc:1403] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.176s user 0.031s sys 0.068s
I20251023 20:15:23.986964 24297 consensus_queue.cc:237] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.987910 23787 raft_consensus.cc:359] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.988178 23787 raft_consensus.cc:385] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.988286 23787 raft_consensus.cc:740] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.988251 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0aeac2f20c4ca48c130a0c851f290f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:23.988565 23787 consensus_queue.cc:260] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.989157 23787 ts_tablet_manager.cc:1434] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.986390 23790 tablet_bootstrap.cc:492] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:23.989538 23787 tablet_bootstrap.cc:492] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:23.989496 23790 ts_tablet_manager.cc:1403] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.143s user 0.022s sys 0.016s
I20251023 20:15:23.991209 23790 raft_consensus.cc:359] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.991321 23787 tablet_bootstrap.cc:654] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:23.991410 23790 raft_consensus.cc:385] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:23.991485 23790 raft_consensus.cc:740] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:23.991691 23790 consensus_queue.cc:260] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:23.992153 23790 ts_tablet_manager.cc:1434] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:23.992455 23790 tablet_bootstrap.cc:492] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:23.986387 23333 leader_election.cc:304] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:23.993508 23333 leader_election.cc:343] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 7301d6121c9d4c8dbc642e895aa4e9f6
W20251023 20:15:23.993772 23333 leader_election.cc:343] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 4ea63540149c4c02abb8ae51209060a3
I20251023 20:15:23.993117 23217 catalog_manager.cc:5649] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
W20251023 20:15:23.994009 23333 leader_election.cc:343] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ef1229b9e0384a82811f244d2bf4c8ce
I20251023 20:15:23.994364 23471 leader_election.cc:304] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:23.994491 23217 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:23.994912 24184 raft_consensus.cc:3060] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:23.996562 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:23.997071 23604 leader_election.cc:343] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.997212 23604 leader_election.cc:304] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:23.997447 24160 raft_consensus.cc:2749] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.994050 24186 raft_consensus.cc:2804] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:23.998150 24184 raft_consensus.cc:2749] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:23.998191 24186 raft_consensus.cc:697] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
W20251023 20:15:23.998544 23333 leader_election.cc:343] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:23.998787 24186 consensus_queue.cc:237] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.000128 23786 ts_tablet_manager.cc:1434] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20251023 20:15:24.001183 23786 tablet_bootstrap.cc:492] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.004577 23787 tablet_bootstrap.cc:492] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.004788 23787 ts_tablet_manager.cc:1403] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20251023 20:15:24.005658 23787 raft_consensus.cc:359] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.005916 23787 raft_consensus.cc:385] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.006055 23787 raft_consensus.cc:740] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.006362 23787 consensus_queue.cc:260] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.006690 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:24.006937 23787 ts_tablet_manager.cc:1434] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:24.009812 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:24.018886 23786 tablet_bootstrap.cc:654] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.022644 23787 tablet_bootstrap.cc:492] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:24.028105 23605 connection.cc:537] server connection from 127.0.0.1:38382 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.028083 23471 connection.cc:537] server connection from 127.0.0.1:44712 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.028671 23949 connection.cc:537] client connection to 127.22.127.189:33509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:23.999087 23333 leader_election.cc:304] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:23.999238 24220 consensus_queue.cc:1048] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:24.028921 23949 leader_election.cc:336] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:24.029070 23949 leader_election.cc:304] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd; no voters: 690ffde75cd745b6a87925967d54471a, ae5a310b4a514c0f812bda133c756062
I20251023 20:15:24.029191 23333 leader_election.cc:304] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.029279 24220 raft_consensus.cc:2804] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.030581 24220 raft_consensus.cc:697] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:24.030790 23333 leader_election.cc:304] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.030870 24220 consensus_queue.cc:237] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.031149 23333 leader_election.cc:304] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.031814 24230 raft_consensus.cc:2804] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.032124 24297 raft_consensus.cc:2804] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.032210 24297 raft_consensus.cc:493] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.032315 24297 raft_consensus.cc:3060] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.033149 23787 tablet_bootstrap.cc:654] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.034998 24297 raft_consensus.cc:515] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:24.035334 23722 connection.cc:537] client connection to 127.22.127.189:33509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:24.035584 24297 leader_election.cc:290] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.035993 23790 tablet_bootstrap.cc:654] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:24.036445 23722 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.127.189:33509)
W20251023 20:15:24.036976 23722 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.22.127.189:33509)
I20251023 20:15:24.037211 24297 raft_consensus.cc:2804] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.037341 24297 raft_consensus.cc:697] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
W20251023 20:15:24.037524 23722 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.22.127.189:33509)
W20251023 20:15:24.038615 23722 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.22.127.189:33509)
I20251023 20:15:24.038823 24297 consensus_queue.cc:237] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.045720 24184 raft_consensus.cc:493] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.045890 24184 raft_consensus.cc:515] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.046542 24184 leader_election.cc:290] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:24.047852 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:24.052695 23470 leader_election.cc:343] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: cff337a6e728489b99fa8c5d4dfece2b
W20251023 20:15:24.060070 23603 connection.cc:537] client connection to 127.22.127.189:33509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.060544 23332 connection.cc:537] client connection to 127.22.127.189:33509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.061722 23203 connection.cc:537] client connection to 127.22.127.189:33509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.061930 23203 leader_election.cc:336] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.062369 23203 leader_election.cc:336] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.062826 23469 connection.cc:537] client connection to 127.22.127.189:33509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:24.069504 23203 consensus_peers.cc:597] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a -> Peer ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509): Couldn't send request to peer ae5a310b4a514c0f812bda133c756062. Status: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:24.077311 24230 raft_consensus.cc:697] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:24.077705 24349 tablet_replica.cc:333] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:24.077847 24230 consensus_queue.cc:237] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.077955 24349 raft_consensus.cc:2243] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:24.078182 24349 raft_consensus.cc:2272] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:24.079263 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 48088.3us
I20251023 20:15:24.084599 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7301d6121c9d4c8dbc642e895aa4e9f6" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.084829 23536 raft_consensus.cc:3060] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:24.087296 23333 connection.cc:537] server connection from 127.0.0.1:47314 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:24.087437 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cff337a6e728489b99fa8c5d4dfece2b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.087570 23536 raft_consensus.cc:2468] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.087663 23397 raft_consensus.cc:2468] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:24.087922 24341 raft_consensus.cc:493] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.088176 24341 raft_consensus.cc:515] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.089012 24341 leader_election.cc:290] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.089200 24220 consensus_queue.cc:1048] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.090201 23787 tablet_bootstrap.cc:492] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.090169 24220 consensus_queue.cc:1048] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.090413 23787 ts_tablet_manager.cc:1403] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.068s user 0.000s sys 0.010s
W20251023 20:15:24.090240 23333 leader_election.cc:343] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 7301d6121c9d4c8dbc642e895aa4e9f6
I20251023 20:15:24.090917 23471 leader_election.cc:304] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.089228 24339 consensus_queue.cc:1048] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.091360 24184 raft_consensus.cc:2804] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.091490 24184 raft_consensus.cc:493] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.091600 24184 raft_consensus.cc:3060] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.091396 24339 consensus_queue.cc:1048] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.091675 24349 ts_tablet_manager.cc:1916] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:24.093642 23786 tablet_bootstrap.cc:492] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.093804 23786 ts_tablet_manager.cc:1403] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.001s
I20251023 20:15:24.094462 24184 raft_consensus.cc:515] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.094677 23786 raft_consensus.cc:359] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.094866 23786 raft_consensus.cc:385] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.094947 23786 raft_consensus.cc:740] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.095172 23786 consensus_queue.cc:260] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.095335 24184 leader_election.cc:290] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:24.095757 23470 leader_election.cc:343] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: cff337a6e728489b99fa8c5d4dfece2b
I20251023 20:15:24.095943 23787 raft_consensus.cc:359] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.097023 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cff337a6e728489b99fa8c5d4dfece2b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.097265 23536 raft_consensus.cc:2393] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:24.097401 23787 raft_consensus.cc:385] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:24.097926 23949 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.189:33509, user_credentials={real_user=slave}} blocked reactor thread for 69016.2us
I20251023 20:15:24.098131 23787 raft_consensus.cc:740] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.098461 23787 consensus_queue.cc:260] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.101909 23333 leader_election.cc:304] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.103173 23216 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_5a52905cde274ed386bff112ef98397a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:24.103960 23333 leader_election.cc:343] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: cff337a6e728489b99fa8c5d4dfece2b
I20251023 20:15:24.104169 23333 leader_election.cc:304] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.104903 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cff337a6e728489b99fa8c5d4dfece2b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:24.104956 23216 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a52905cde274ed386bff112ef98397a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.104974 24220 raft_consensus.cc:2804] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.105118 23397 raft_consensus.cc:3060] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.105162 24220 raft_consensus.cc:697] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:24.105665 24220 consensus_queue.cc:237] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.108079 23397 raft_consensus.cc:2468] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:24.108242 24297 raft_consensus.cc:2749] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.108783 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_5e2be9d05d7d431685748f79ce1c7311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:24.109031 23214 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_8c22652456c84553954550e1018e83e5" } modify_external_catalogs: true
I20251023 20:15:24.109484 23212 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_14cd5c04c2b84209a82106f3424a8bd0" } modify_external_catalogs: true
W20251023 20:15:24.109622 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e2be9d05d7d431685748f79ce1c7311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.110025 23213 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_532d7e6d0eec4f09bcdf525c353fe0a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:24.110143 23210 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_4f8238f0492b4364b731c0bf5e2bd53a" } new_table_name: "default.Table_bfabdf6a28e645d697a566e89f2d79e3" modify_external_catalogs: true
W20251023 20:15:24.110739 23213 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_532d7e6d0eec4f09bcdf525c353fe0a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.111099 23211 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_20216f4bd8cd4150bfb55aa1e918e7c7" } modify_external_catalogs: true
I20251023 20:15:24.118230 23790 tablet_bootstrap.cc:492] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:24.118402 23790 ts_tablet_manager.cc:1403] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.126s user 0.007s sys 0.002s
I20251023 20:15:24.118604 24259 raft_consensus.cc:493] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.118767 24259 raft_consensus.cc:515] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.119263 23790 raft_consensus.cc:359] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.119442 23790 raft_consensus.cc:385] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.119468 24259 leader_election.cc:290] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:24.119536 23790 raft_consensus.cc:740] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:24.121752 23470 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 83da09ee0d10435ba89a19dc378d34eb
W20251023 20:15:24.122427 23470 consensus_peers.cc:597] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e418a62d00824cff89b37a7bace5c469. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:24.124191 23790 consensus_queue.cc:260] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.124755 23790 ts_tablet_manager.cc:1434] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:24.127015 23787 ts_tablet_manager.cc:1434] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20251023 20:15:24.127358 23787 tablet_bootstrap.cc:492] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.128974 24259 consensus_queue.cc:1048] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.130540 23471 leader_election.cc:304] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.130697 24174 raft_consensus.cc:493] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.130896 24174 raft_consensus.cc:515] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.131103 24259 raft_consensus.cc:2804] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.131234 24259 raft_consensus.cc:697] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:24.131505 24174 leader_election.cc:290] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.131501 24259 consensus_queue.cc:237] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.132639 23790 tablet_bootstrap.cc:492] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:24.134167 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a21ec49194e24723b31fd9c86f1f85e4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.134176 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a21ec49194e24723b31fd9c86f1f85e4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:24.134792 23605 leader_election.cc:343] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:24.135358 23604 leader_election.cc:343] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.135484 23604 leader_election.cc:304] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.136041 24174 raft_consensus.cc:2749] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.136430 24174 raft_consensus.cc:493] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.136583 24174 raft_consensus.cc:515] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.137216 24174 leader_election.cc:290] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.138427 23790 tablet_bootstrap.cc:654] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.138532 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "523d72012c364457b9a2a299f2f789a1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:24.139392 23605 leader_election.cc:343] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.140491 23787 tablet_bootstrap.cc:654] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.140514 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "523d72012c364457b9a2a299f2f789a1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.141068 23604 leader_election.cc:343] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.141227 23604 leader_election.cc:304] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.141335 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:24.141705 24174 raft_consensus.cc:2749] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.095801 23786 ts_tablet_manager.cc:1434] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:24.142819 23786 tablet_bootstrap.cc:492] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.143657 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83da09ee0d10435ba89a19dc378d34eb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.143890 23397 raft_consensus.cc:2468] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:24.144522 23786 tablet_bootstrap.cc:654] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.146456 23471 leader_election.cc:304] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.147177 24259 raft_consensus.cc:2804] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.147486 24259 raft_consensus.cc:493] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.147773 24259 raft_consensus.cc:3060] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.155124 24259 raft_consensus.cc:515] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:24.156273 24345 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:15:24.041262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.127.189:33509 (local address 127.0.0.1:51808)
1023 20:15:24.137494 (+ 96232us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:24.137895 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":95919,"thread_start_us":37198,"threads_started":1}
I20251023 20:15:24.157027 24259 leader_election.cc:290] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:24.157366 23470 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:24.158016 23787 tablet_bootstrap.cc:492] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.158238 23787 ts_tablet_manager.cc:1403] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.007s
I20251023 20:15:24.159124 23787 raft_consensus.cc:359] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.159389 23787 raft_consensus.cc:385] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.159531 23787 raft_consensus.cc:740] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.160619 23787 consensus_queue.cc:260] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.161331 23787 ts_tablet_manager.cc:1434] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:24.163825 24339 raft_consensus.cc:493] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.164065 24339 raft_consensus.cc:515] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.164906 24339 leader_election.cc:290] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.167277 23214 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_8c22652456c84553954550e1018e83e5" } modify_external_catalogs: true
I20251023 20:15:24.167637 23211 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_20216f4bd8cd4150bfb55aa1e918e7c7" } modify_external_catalogs: true
I20251023 20:15:24.169032 23790 tablet_bootstrap.cc:492] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:24.169622 23790 ts_tablet_manager.cc:1403] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.001s
I20251023 20:15:24.170742 23790 raft_consensus.cc:359] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.172016 23790 raft_consensus.cc:385] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.172289 23790 raft_consensus.cc:740] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.172551 23212 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_14cd5c04c2b84209a82106f3424a8bd0" } modify_external_catalogs: true
I20251023 20:15:24.172683 23790 consensus_queue.cc:260] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.173802 23787 tablet_bootstrap.cc:492] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.174111 24230 raft_consensus.cc:493] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.174271 23790 ts_tablet_manager.cc:1434] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:24.176440 23787 tablet_bootstrap.cc:654] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.176901 23790 tablet_bootstrap.cc:492] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:24.179065 23790 tablet_bootstrap.cc:654] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.184171 24259 raft_consensus.cc:493] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.184350 24259 raft_consensus.cc:515] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.185467 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2dbb7997d0f4e6bac48cfb196f5ccb6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.174343 24230 raft_consensus.cc:515] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.185925 24186 consensus_queue.cc:1048] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.186627 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edec01a63a3046d9ae5a372fc9068b52" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.186787 24230 leader_election.cc:290] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.189169 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edec01a63a3046d9ae5a372fc9068b52" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.189375 23668 raft_consensus.cc:2468] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.201781 24319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690ffde75cd745b6a87925967d54471a. Latest consensus state: current_term: 2 leader_uuid: "690ffde75cd745b6a87925967d54471a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:24.202157 24319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:24.202255 24338 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:24.212162 24259 leader_election.cc:290] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:24.216508 24259 raft_consensus.cc:493] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.216733 24259 raft_consensus.cc:515] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.217587 24259 leader_election.cc:290] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:24.217976 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.218245 23668 raft_consensus.cc:2393] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:24.218659 24363 tablet_replica.cc:333] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:24.218986 24363 raft_consensus.cc:2243] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:24.219359 24363 pending_rounds.cc:70] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Trying to abort 1 pending ops.
I20251023 20:15:24.219465 24363 pending_rounds.cc:77] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082146216136704 op_type: NO_OP noop_request { }
I20251023 20:15:24.219659 24363 raft_consensus.cc:2889] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:24.219794 24363 raft_consensus.cc:2272] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:24.231603 23787 tablet_bootstrap.cc:492] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.231830 23787 ts_tablet_manager.cc:1403] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.004s
I20251023 20:15:24.232728 23787 raft_consensus.cc:359] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.232932 23787 raft_consensus.cc:385] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.233007 23787 raft_consensus.cc:740] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.233318 23787 consensus_queue.cc:260] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.234102 23787 ts_tablet_manager.cc:1434] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:24.234489 23787 tablet_bootstrap.cc:492] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.236238 23787 tablet_bootstrap.cc:654] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.237872 24338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "690ffde75cd745b6a87925967d54471a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:24.238349 24338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:24.242960 24259 raft_consensus.cc:493] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.243130 24259 raft_consensus.cc:515] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.243232 24339 raft_consensus.cc:493] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.243974 24259 leader_election.cc:290] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.243991 24339 raft_consensus.cc:515] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.244596 24339 leader_election.cc:290] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.244753 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83da09ee0d10435ba89a19dc378d34eb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:24.244969 23399 raft_consensus.cc:3060] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.247924 23399 raft_consensus.cc:2468] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:24.248864 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.249207 23670 raft_consensus.cc:2393] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:24.250916 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d1ddde7e3134e23b3a9356523a93753" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.251068 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2dbb7997d0f4e6bac48cfb196f5ccb6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.251401 23399 raft_consensus.cc:2393] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:24.253244 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52bc660b9da54c5d984381a267e1efa2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.253463 23533 raft_consensus.cc:2468] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.256537 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a27f192792cf41288d69ca15ab538472" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.256794 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a27f192792cf41288d69ca15ab538472" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.257012 23668 raft_consensus.cc:2468] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.257642 23471 leader_election.cc:304] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.257915 24259 raft_consensus.cc:2804] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.258175 24259 raft_consensus.cc:697] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:24.258626 23471 leader_election.cc:304] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:24.259032 24184 raft_consensus.cc:3060] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.259150 24259 consensus_queue.cc:237] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.262040 24184 raft_consensus.cc:2749] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.262636 23471 leader_election.cc:304] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:24.263475 23333 leader_election.cc:304] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.263733 24184 raft_consensus.cc:3060] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.264621 23786 tablet_bootstrap.cc:492] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.264763 23786 ts_tablet_manager.cc:1403] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.122s user 0.002s sys 0.019s
I20251023 20:15:24.265756 23786 raft_consensus.cc:359] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.265961 23786 raft_consensus.cc:385] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.266049 23786 raft_consensus.cc:740] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.266258 23786 consensus_queue.cc:260] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.266608 24184 raft_consensus.cc:2749] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.268569 24339 raft_consensus.cc:2804] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.268680 24339 raft_consensus.cc:493] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.268771 24339 raft_consensus.cc:3060] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.271486 24363 ts_tablet_manager.cc:1916] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:24.271601 24339 raft_consensus.cc:515] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.272253 24339 leader_election.cc:290] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:24.272763 23333 leader_election.cc:343] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 52bc660b9da54c5d984381a267e1efa2
I20251023 20:15:24.273150 23333 leader_election.cc:304] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.273618 24230 raft_consensus.cc:2804] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.273726 24230 raft_consensus.cc:493] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.273797 24230 raft_consensus.cc:3060] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.276409 24230 raft_consensus.cc:515] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.277069 24230 leader_election.cc:290] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.267261 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7645c36c983f438f8b1223256fd3ad2c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:24.277935 23333 leader_election.cc:343] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.279294 23333 leader_election.cc:304] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.279690 24230 raft_consensus.cc:2804] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.279901 24230 raft_consensus.cc:493] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.280067 24230 raft_consensus.cc:3060] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.282783 24230 raft_consensus.cc:515] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.283535 24230 leader_election.cc:290] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.283586 24297 raft_consensus.cc:493] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.283885 24297 raft_consensus.cc:515] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.286829 24349 ts_tablet_manager.cc:1929] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:24.286972 24297 leader_election.cc:290] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.287011 24349 log.cc:1199] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/f4778436aed64b53adb2bab54bcf263b
I20251023 20:15:24.287408 24349 ts_tablet_manager.cc:1950] T f4778436aed64b53adb2bab54bcf263b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:24.287694 23333 leader_election.cc:343] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.288761 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet f4778436aed64b53adb2bab54bcf263b (table default.Table_5abd17a4558f4f81a956cdf6df0d4524 [id=46e81c2c4c164452980264259fcff377]) successfully deleted
I20251023 20:15:24.290629 23786 ts_tablet_manager.cc:1434] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.026s user 0.000s sys 0.005s
I20251023 20:15:24.291026 23786 tablet_bootstrap.cc:492] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.292713 24174 raft_consensus.cc:493] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.292922 24174 raft_consensus.cc:515] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.293637 23786 tablet_bootstrap.cc:654] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.293628 24174 leader_election.cc:290] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.294456 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a2f3adb5d347a0adc198cd1ef93304" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.294692 23398 raft_consensus.cc:2468] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.294937 24265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "690ffde75cd745b6a87925967d54471a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:24.295190 24265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:24.295255 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a2f3adb5d347a0adc198cd1ef93304" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.295421 23605 leader_election.cc:304] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.295801 24160 raft_consensus.cc:2804] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251023 20:15:24.295866 23604 leader_election.cc:343] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.295938 24174 raft_consensus.cc:493] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.296056 24174 raft_consensus.cc:515] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.296650 24174 leader_election.cc:290] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.295908 24160 raft_consensus.cc:493] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.298532 24160 raft_consensus.cc:3060] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.301584 23790 tablet_bootstrap.cc:492] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:24.301610 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edec01a63a3046d9ae5a372fc9068b52" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.302155 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8289ce37fa074891beb584bf2fff9747" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.302374 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edec01a63a3046d9ae5a372fc9068b52" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:24.302554 23668 raft_consensus.cc:3060] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:24.303066 23604 leader_election.cc:343] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.303246 24160 raft_consensus.cc:515] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.303917 24160 leader_election.cc:290] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.304517 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a2f3adb5d347a0adc198cd1ef93304" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:24.305011 23604 leader_election.cc:343] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.305930 23668 raft_consensus.cc:2468] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.308198 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a27f192792cf41288d69ca15ab538472" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:24.308627 23668 raft_consensus.cc:3060] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.310463 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a27f192792cf41288d69ca15ab538472" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.312094 24160 consensus_queue.cc:1048] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.313992 23787 tablet_bootstrap.cc:492] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.314134 23787 ts_tablet_manager.cc:1403] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.016s
W20251023 20:15:24.314482 23217 catalog_manager.cc:5101] could not convert tablet's e5368813164949ff80e27bf59dcc931a schema from PB: OK
I20251023 20:15:24.314987 23787 raft_consensus.cc:359] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.315336 23787 raft_consensus.cc:385] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.315435 23787 raft_consensus.cc:740] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.315665 23787 consensus_queue.cc:260] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.316180 23787 ts_tablet_manager.cc:1434] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:24.316651 23787 tablet_bootstrap.cc:492] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.318610 23787 tablet_bootstrap.cc:654] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.319558 23668 raft_consensus.cc:2468] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.320760 23786 tablet_bootstrap.cc:492] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.320941 23786 ts_tablet_manager.cc:1403] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20251023 20:15:24.321805 23786 raft_consensus.cc:359] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.322032 23786 raft_consensus.cc:385] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.322108 23786 raft_consensus.cc:740] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.322367 23786 consensus_queue.cc:260] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.323704 23471 leader_election.cc:304] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:24.324074 24259 raft_consensus.cc:3060] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.326769 24259 raft_consensus.cc:2749] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.331027 24363 ts_tablet_manager.cc:1929] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:24.331216 24363 log.cc:1199] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:24.302019 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8289ce37fa074891beb584bf2fff9747" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.306525 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a2f3adb5d347a0adc198cd1ef93304" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:24.315716 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:24.332017 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52bc660b9da54c5d984381a267e1efa2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.332182 23533 raft_consensus.cc:3060] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.332190 23399 raft_consensus.cc:3060] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.332219 23397 raft_consensus.cc:2468] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.332757 23605 leader_election.cc:304] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.334384 24174 raft_consensus.cc:2804] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.334512 24174 raft_consensus.cc:493] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.334604 24174 raft_consensus.cc:3060] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.335008 23533 raft_consensus.cc:2468] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.338615 24160 consensus_queue.cc:1048] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.339838 24174 raft_consensus.cc:515] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.340881 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8289ce37fa074891beb584bf2fff9747" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:24.341218 23396 raft_consensus.cc:3060] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.341396 23786 ts_tablet_manager.cc:1434] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.020s user 0.000s sys 0.003s
I20251023 20:15:24.344280 23396 raft_consensus.cc:2468] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.345252 23605 leader_election.cc:304] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.346403 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8289ce37fa074891beb584bf2fff9747" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.346500 24160 raft_consensus.cc:2804] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.346683 24160 raft_consensus.cc:697] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:24.346952 24160 consensus_queue.cc:237] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:24.347009 23604 leader_election.cc:343] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.347194 24174 leader_election.cc:290] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.350564 23399 raft_consensus.cc:2468] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.351104 23605 leader_election.cc:304] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.351467 24198 raft_consensus.cc:2804] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.351581 24198 raft_consensus.cc:697] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:24.351867 24198 consensus_queue.cc:237] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.353837 23786 tablet_bootstrap.cc:492] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.354177 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "283166bbd8004ce3bca9f2ecbdbcbf48" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.354420 23533 raft_consensus.cc:2468] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.354892 23790 ts_tablet_manager.cc:1403] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.178s user 0.007s sys 0.009s
I20251023 20:15:24.355701 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "283166bbd8004ce3bca9f2ecbdbcbf48" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.355983 23668 raft_consensus.cc:2468] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.358196 24363 ts_tablet_manager.cc:1950] T 4beebee06e5341a29c65d0c3cf985ef4 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:24.359640 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 4beebee06e5341a29c65d0c3cf985ef4 (table default.Table_3220c87e07554676ba34368830409e89 [id=85bda6747ffe49f1b5a123f41013c426]) successfully deleted
I20251023 20:15:24.359614 23790 raft_consensus.cc:359] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.359804 23790 raft_consensus.cc:385] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.359906 23790 raft_consensus.cc:740] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.360143 23790 consensus_queue.cc:260] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.362241 24383 consensus_queue.cc:1048] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.366616 24198 consensus_queue.cc:1048] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.369720 23786 tablet_bootstrap.cc:654] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.373651 24198 consensus_queue.cc:1048] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.378479 23786 tablet_bootstrap.cc:492] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.379340 23786 ts_tablet_manager.cc:1403] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20251023 20:15:24.380326 23786 raft_consensus.cc:359] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.381467 23786 raft_consensus.cc:385] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.381723 23786 raft_consensus.cc:740] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.382124 23786 consensus_queue.cc:260] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.386420 24174 raft_consensus.cc:493] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.386651 24174 raft_consensus.cc:515] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.387408 24174 leader_election.cc:290] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.387943 23786 ts_tablet_manager.cc:1434] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:24.387899 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7540354f8577482785e6341fa9022315" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.388818 23604 leader_election.cc:343] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.390393 23786 tablet_bootstrap.cc:492] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.391228 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7540354f8577482785e6341fa9022315" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.391538 23399 raft_consensus.cc:2468] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.391981 23605 leader_election.cc:304] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.392212 23786 tablet_bootstrap.cc:654] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.392457 24174 raft_consensus.cc:2804] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.397202 24160 raft_consensus.cc:493] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.397418 24160 raft_consensus.cc:515] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.398474 24160 leader_election.cc:290] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.398576 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed46b7e67e4e4ae7b30ea69f1d3fdc2f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.399119 23604 leader_election.cc:343] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.399427 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed46b7e67e4e4ae7b30ea69f1d3fdc2f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.399686 23399 raft_consensus.cc:2468] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.397176 24174 raft_consensus.cc:493] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.400097 24174 raft_consensus.cc:3060] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.400167 23605 leader_election.cc:304] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.400589 24160 raft_consensus.cc:2804] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.400756 24160 raft_consensus.cc:493] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.400905 24160 raft_consensus.cc:3060] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.403446 23786 tablet_bootstrap.cc:492] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.403395 24174 raft_consensus.cc:515] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.403457 24160 raft_consensus.cc:515] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.403647 23786 ts_tablet_manager.cc:1403] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20251023 20:15:24.404348 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7540354f8577482785e6341fa9022315" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:24.404392 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7540354f8577482785e6341fa9022315" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.404551 23396 raft_consensus.cc:3060] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.404563 23786 raft_consensus.cc:359] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.404759 23786 raft_consensus.cc:385] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.404824 23786 raft_consensus.cc:740] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:24.404906 23604 leader_election.cc:343] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.405117 24174 leader_election.cc:290] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.407294 23396 raft_consensus.cc:2468] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.407878 23605 leader_election.cc:304] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.408404 24174 raft_consensus.cc:2804] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.408859 24174 raft_consensus.cc:697] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:24.409538 24174 consensus_queue.cc:237] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.411383 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed46b7e67e4e4ae7b30ea69f1d3fdc2f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:24.411600 23396 raft_consensus.cc:3060] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.414397 23396 raft_consensus.cc:2468] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.414913 23605 leader_election.cc:304] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.415283 24198 raft_consensus.cc:2804] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.405067 23786 consensus_queue.cc:260] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.417356 23786 ts_tablet_manager.cc:1434] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20251023 20:15:24.418886 23786 tablet_bootstrap.cc:492] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.421902 24400 raft_consensus.cc:493] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.422118 24400 raft_consensus.cc:515] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.423197 24400 leader_election.cc:290] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.423422 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.424338 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.424829 23604 leader_election.cc:343] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:24.425222 23605 leader_election.cc:343] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.425299 24198 raft_consensus.cc:697] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:24.425362 23605 leader_election.cc:304] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.425711 24198 consensus_queue.cc:237] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.425882 24401 raft_consensus.cc:2749] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.426855 23217 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_37c825c402604e08a862951e693de880" } modify_external_catalogs: true
I20251023 20:15:24.427053 23217 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_37c825c402604e08a862951e693de880" } modify_external_catalogs: true
I20251023 20:15:24.427145 23213 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_dcd3e2d5526c40bfb65ac0f8188c4d48" } new_table_name: "default.Table_1d7f65ff02bb4fccb1a45097c5036126" modify_external_catalogs: true
I20251023 20:15:24.427637 23216 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_a9bddf9d668f414dae97b08900de3f65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:24.428016 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed46b7e67e4e4ae7b30ea69f1d3fdc2f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:24.428145 23216 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9bddf9d668f414dae97b08900de3f65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:24.428475 23604 leader_election.cc:343] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.428681 24160 leader_election.cc:290] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.428720 23215 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_1f76833d42104cee91c1406f649fa028" } new_table_name: "default.Table_e2bc01c543da4117a80827430a7d66a5" modify_external_catalogs: true
I20251023 20:15:24.435920 23790 ts_tablet_manager.cc:1434] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.077s user 0.002s sys 0.000s
I20251023 20:15:24.436236 23790 tablet_bootstrap.cc:492] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:24.442119 24186 raft_consensus.cc:493] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.442291 24186 raft_consensus.cc:515] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.442884 24186 leader_election.cc:290] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.444422 23216 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_cd166f9dbcdb47ad988e2d387a6a1c5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:24.444972 23216 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd166f9dbcdb47ad988e2d387a6a1c5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.445850 23786 tablet_bootstrap.cc:654] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:24.449654 23470 consensus_peers.cc:597] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8da82e6d24b14a129b91bb5298139d07. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:24.465638 23790 tablet_bootstrap.cc:654] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.466764 23786 tablet_bootstrap.cc:492] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.466938 23786 ts_tablet_manager.cc:1403] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
I20251023 20:15:24.467573 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:24.467705 23786 raft_consensus.cc:359] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.467816 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:24.467895 23786 raft_consensus.cc:385] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.467967 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:24.467984 23786 raft_consensus.cc:740] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.468191 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:24.468227 23786 consensus_queue.cc:260] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.468598 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:24.469101 23786 ts_tablet_manager.cc:1434] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:24.469584 23786 tablet_bootstrap.cc:492] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.470337 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:24.470412 23787 tablet_bootstrap.cc:492] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.470595 23787 ts_tablet_manager.cc:1403] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.154s user 0.030s sys 0.035s
I20251023 20:15:24.470597 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:24.470981 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:24.471285 23786 tablet_bootstrap.cc:654] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.471371 23787 raft_consensus.cc:359] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.471580 23787 raft_consensus.cc:385] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.471659 23787 raft_consensus.cc:740] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.471873 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]). Expected version 1 got 0
I20251023 20:15:24.471913 23787 consensus_queue.cc:260] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.472393 23787 ts_tablet_manager.cc:1434] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:24.472633 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:24.472711 23787 tablet_bootstrap.cc:492] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.473452 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:24.473721 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:24.474365 23787 tablet_bootstrap.cc:654] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.474576 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:24.474828 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:24.475266 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:24.479063 23786 tablet_bootstrap.cc:492] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.479218 23786 ts_tablet_manager.cc:1403] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20251023 20:15:24.479447 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:24.479681 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:24.480073 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]). Expected version 3 got 0
I20251023 20:15:24.480187 23786 raft_consensus.cc:359] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.480368 23786 raft_consensus.cc:385] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.480448 23786 raft_consensus.cc:740] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.480571 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:24.480717 23786 consensus_queue.cc:260] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.480787 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:24.481465 23786 ts_tablet_manager.cc:1434] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:24.481134 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:24.481956 23786 tablet_bootstrap.cc:492] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.483000 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:24.483223 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:24.483492 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:24.483716 23786 tablet_bootstrap.cc:654] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.484115 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:24.484397 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 1 got 0
I20251023 20:15:24.486137 23787 tablet_bootstrap.cc:492] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.486342 23787 ts_tablet_manager.cc:1403] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20251023 20:15:24.487152 23787 raft_consensus.cc:359] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.487406 23787 raft_consensus.cc:385] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.487532 23787 raft_consensus.cc:740] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.487854 23787 consensus_queue.cc:260] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.488581 23787 ts_tablet_manager.cc:1434] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:24.488968 23787 tablet_bootstrap.cc:492] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.491364 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:24.491688 23333 leader_election.cc:304] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.492148 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:24.493945 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]). Expected version 2 got 0
I20251023 20:15:24.497359 24341 raft_consensus.cc:2804] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.497525 24341 raft_consensus.cc:697] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:24.498178 23333 leader_election.cc:304] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.498473 24339 raft_consensus.cc:2804] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.498625 24339 raft_consensus.cc:697] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
W20251023 20:15:24.499071 23333 leader_election.cc:343] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 52bc660b9da54c5d984381a267e1efa2
I20251023 20:15:24.499316 23333 leader_election.cc:304] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.501119 23787 tablet_bootstrap.cc:654] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.501783 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]). Expected version 2 got 0
I20251023 20:15:24.499626 24341 consensus_queue.cc:237] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.503540 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
I20251023 20:15:24.503743 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:24.503931 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:24.504122 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]). Expected version 1 got 0
I20251023 20:15:24.499963 24339 consensus_queue.cc:237] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.504482 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]). Expected version 1 got 0
I20251023 20:15:24.505321 23219 catalog_manager.cc:5649] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:24.505962 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]). Expected version 4 got 0
I20251023 20:15:24.506546 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]). Expected version 2 got 0
I20251023 20:15:24.507499 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:24.507908 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:24.508170 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971]). Expected version 1 got 0
I20251023 20:15:24.508517 23787 tablet_bootstrap.cc:492] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.508713 23787 ts_tablet_manager.cc:1403] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20251023 20:15:24.508944 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:24.509179 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:24.509495 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:24.509578 23787 raft_consensus.cc:359] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.509814 23787 raft_consensus.cc:385] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.509785 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]). Expected version 3 got 0
I20251023 20:15:24.509923 23787 raft_consensus.cc:740] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.510089 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:24.510190 23787 consensus_queue.cc:260] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.510699 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:24.511133 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:24.511165 23787 ts_tablet_manager.cc:1434] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:24.511487 23787 tablet_bootstrap.cc:492] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.511696 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:24.513278 23787 tablet_bootstrap.cc:654] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.515148 23786 tablet_bootstrap.cc:492] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.515410 23786 ts_tablet_manager.cc:1403] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.015s
I20251023 20:15:24.516351 23786 raft_consensus.cc:359] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.516614 23786 raft_consensus.cc:385] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.516741 23786 raft_consensus.cc:740] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.517061 23786 consensus_queue.cc:260] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.521543 23786 ts_tablet_manager.cc:1434] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:24.521934 23786 tablet_bootstrap.cc:492] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.524431 23786 tablet_bootstrap.cc:654] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.527702 24383 consensus_queue.cc:1048] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.540973 24186 raft_consensus.cc:2804] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.541123 24186 raft_consensus.cc:493] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.541226 24186 raft_consensus.cc:3060] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.544010 24186 raft_consensus.cc:515] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.544703 24186 leader_election.cc:290] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:24.545186 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 45875.2us
W20251023 20:15:24.545945 23333 consensus_peers.cc:597] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fc7e18054c440098dc2470143ea7fae. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:24.546276 23333 leader_election.cc:343] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:24.546646 23333 leader_election.cc:343] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.547006 23333 leader_election.cc:304] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:24.547290 24186 raft_consensus.cc:2804] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.547466 24186 raft_consensus.cc:697] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:24.547771 24186 consensus_queue.cc:237] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.553478 23787 tablet_bootstrap.cc:492] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.553649 23787 ts_tablet_manager.cc:1403] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.002s
I20251023 20:15:24.554486 23787 raft_consensus.cc:359] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.554697 23787 raft_consensus.cc:385] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.554785 23787 raft_consensus.cc:740] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.555042 23787 consensus_queue.cc:260] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.555527 23787 ts_tablet_manager.cc:1434] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:24.555862 23787 tablet_bootstrap.cc:492] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.557667 23787 tablet_bootstrap.cc:654] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.561426 23790 tablet_bootstrap.cc:492] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:24.561674 23790 ts_tablet_manager.cc:1403] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20251023 20:15:24.562542 23790 raft_consensus.cc:359] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.563373 23790 raft_consensus.cc:385] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.569365 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ae19cc80194363b8a9c96fbafb4ebe" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.569743 23532 raft_consensus.cc:2468] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.570801 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ae19cc80194363b8a9c96fbafb4ebe" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.571062 23668 raft_consensus.cc:2468] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.572687 24383 consensus_queue.cc:1048] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.576694 23787 tablet_bootstrap.cc:492] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.576896 23787 ts_tablet_manager.cc:1403] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251023 20:15:24.577767 23787 raft_consensus.cc:359] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.578004 23787 raft_consensus.cc:385] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.578076 23787 raft_consensus.cc:740] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.578354 23787 consensus_queue.cc:260] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.579020 23787 ts_tablet_manager.cc:1434] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:24.579344 23787 tablet_bootstrap.cc:492] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.580700 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:24.580973 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:24.581230 23787 tablet_bootstrap.cc:654] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.581245 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:24.581693 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:24.581880 23790 raft_consensus.cc:740] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.581986 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:24.582211 23790 consensus_queue.cc:260] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.582291 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:24.582751 23790 ts_tablet_manager.cc:1434] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251023 20:15:24.582795 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:24.583098 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:24.583463 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:24.583626 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:24.583801 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:24.583971 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:24.584148 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:24.584368 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:24.584581 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:24.584751 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]). Expected version 1 got 0
I20251023 20:15:24.585318 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:24.585507 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:24.585686 23790 tablet_bootstrap.cc:492] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:24.585927 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:24.586302 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:24.586524 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:24.586691 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:24.587028 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:24.587302 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:24.587416 23790 tablet_bootstrap.cc:654] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.587683 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:24.587849 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:24.588153 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:24.588339 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
I20251023 20:15:24.588577 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:24.588850 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:24.589185 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:24.589332 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 1 got 0
I20251023 20:15:24.589946 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:24.590329 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]). Expected version 1 got 0
I20251023 20:15:24.590648 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:24.590826 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]). Expected version 1 got 0
I20251023 20:15:24.591281 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]). Expected version 3 got 0
I20251023 20:15:24.591455 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971]). Expected version 1 got 0
I20251023 20:15:24.591780 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:24.592170 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:24.592381 23218 catalog_manager.cc:5649] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:24.592829 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]). Expected version 2 got 0
I20251023 20:15:24.593268 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]). Expected version 2 got 0
I20251023 20:15:24.593595 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:24.593802 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]). Expected version 3 got 0
I20251023 20:15:24.593976 23786 tablet_bootstrap.cc:492] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.593997 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]). Expected version 4 got 0
I20251023 20:15:24.594147 23786 ts_tablet_manager.cc:1403] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.023s
I20251023 20:15:24.594936 23786 raft_consensus.cc:359] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.595036 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:24.595137 23786 raft_consensus.cc:385] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.595212 23786 raft_consensus.cc:740] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.595476 23786 consensus_queue.cc:260] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.596020 23786 ts_tablet_manager.cc:1434] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:24.596156 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:24.596333 23786 tablet_bootstrap.cc:492] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.596458 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]). Expected version 2 got 0
I20251023 20:15:24.596906 23211 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 5acdc66df9c748a1a6300962f62fcf36
W20251023 20:15:24.597019 23604 consensus_peers.cc:597] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4778436aed64b53adb2bab54bcf263b. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:24.597846 23211 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_2dc2a5b423424735a1c370afde321275"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:24.598009 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:24.598412 23211 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dc2a5b423424735a1c370afde321275 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.598799 23786 tablet_bootstrap.cc:654] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:24.598874 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
I20251023 20:15:24.600018 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:24.600968 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
I20251023 20:15:24.606853 24198 raft_consensus.cc:493] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.607024 24198 raft_consensus.cc:515] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.607671 24198 leader_election.cc:290] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.608640 23211 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_f06b4a8de0ed4fee83a2d7e2a2eb3823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:24.608780 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e3d422a66f432789c0cf4a00e57af2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:24.609297 23605 leader_election.cc:343] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:24.609512 23211 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f06b4a8de0ed4fee83a2d7e2a2eb3823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.611284 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e3d422a66f432789c0cf4a00e57af2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.611857 23604 leader_election.cc:343] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.611982 23604 leader_election.cc:304] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.612366 24198 raft_consensus.cc:2749] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.617449 23211 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_44c4bece35c24b7b9eae08ab8b835c78" } new_table_name: "default.Table_519888eb86d3490baf10c0d615eea255" modify_external_catalogs: true
I20251023 20:15:24.618875 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "283166bbd8004ce3bca9f2ecbdbcbf48" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.619184 23533 raft_consensus.cc:3060] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.622229 23533 raft_consensus.cc:2468] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.623505 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "283166bbd8004ce3bca9f2ecbdbcbf48" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:24.623695 23670 raft_consensus.cc:3060] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.626585 23670 raft_consensus.cc:2468] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.639708 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.189:33509
--webserver_interface=127.22.127.189
--webserver_port=42879
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251023 20:15:24.641315 23333 leader_election.cc:304] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.642158 24186 raft_consensus.cc:2804] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.642248 24186 raft_consensus.cc:493] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.642340 24186 raft_consensus.cc:3060] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.645155 24186 raft_consensus.cc:515] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.645874 24186 leader_election.cc:290] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.646384 23333 leader_election.cc:304] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.647127 24186 raft_consensus.cc:2804] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.647297 24186 raft_consensus.cc:697] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:24.647774 24186 consensus_queue.cc:237] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.649718 24186 consensus_queue.cc:1048] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.651938 24198 raft_consensus.cc:493] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.652040 24174 consensus_queue.cc:1048] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.652091 24198 raft_consensus.cc:515] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.653177 24198 leader_election.cc:290] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.654042 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cec51c54ea534b1da4b1d170e79d9f1c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.654240 23398 raft_consensus.cc:2468] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.654771 23605 leader_election.cc:304] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.655040 24198 raft_consensus.cc:2804] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.655158 24198 raft_consensus.cc:493] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.655234 24198 raft_consensus.cc:3060] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.657939 24198 raft_consensus.cc:515] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.658943 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cec51c54ea534b1da4b1d170e79d9f1c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:24.659024 24198 leader_election.cc:290] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.659124 23398 raft_consensus.cc:3060] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.659291 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cec51c54ea534b1da4b1d170e79d9f1c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:24.659780 23604 leader_election.cc:343] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.662177 23398 raft_consensus.cc:2468] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.662693 23605 leader_election.cc:304] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.663012 24198 raft_consensus.cc:2804] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.663362 24198 raft_consensus.cc:697] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:24.663895 24198 consensus_queue.cc:237] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.664523 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cec51c54ea534b1da4b1d170e79d9f1c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.665504 23604 leader_election.cc:343] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:24.670964 23333 consensus_peers.cc:597] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:24.679258 24160 raft_consensus.cc:493] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.679469 24160 raft_consensus.cc:515] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.680131 24160 leader_election.cc:290] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.687189 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0438593fbbdf47cebfb64121acad104a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.687527 23533 raft_consensus.cc:2468] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.688038 23604 leader_election.cc:304] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:24.688336 24160 raft_consensus.cc:2804] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.688640 24160 raft_consensus.cc:493] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.688791 24160 raft_consensus.cc:3060] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.692612 24160 raft_consensus.cc:515] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.681016 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0438593fbbdf47cebfb64121acad104a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.693365 24160 leader_election.cc:290] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:24.694286 23605 leader_election.cc:343] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.694821 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0438593fbbdf47cebfb64121acad104a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:24.695396 23605 leader_election.cc:343] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.696043 24186 consensus_queue.cc:1048] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.696545 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0438593fbbdf47cebfb64121acad104a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.696803 23533 raft_consensus.cc:3060] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.698869 23634 tablet_service.cc:1505] Processing CreateTablet for tablet ea3149f906064f8c8167292ec86540fd (DEFAULT_TABLE table=default.Table_532d7e6d0eec4f09bcdf525c353fe0a8 [id=1c95889d5d3d408c9508b0d4509c84d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.699321 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3149f906064f8c8167292ec86540fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.708302 23516 tablet_service.cc:1505] Processing CreateTablet for tablet ea3149f906064f8c8167292ec86540fd (DEFAULT_TABLE table=default.Table_532d7e6d0eec4f09bcdf525c353fe0a8 [id=1c95889d5d3d408c9508b0d4509c84d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.708844 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3149f906064f8c8167292ec86540fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.714790 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ae19cc80194363b8a9c96fbafb4ebe" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.715035 23532 raft_consensus.cc:3060] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.718011 23532 raft_consensus.cc:2468] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.721710 23634 tablet_service.cc:1505] Processing CreateTablet for tablet de2ead20b8fd4053af7cbc69cd247413 (DEFAULT_TABLE table=default.Table_5a52905cde274ed386bff112ef98397a [id=1674ccf219e44d6e84ad9f62755ff1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.722221 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2ead20b8fd4053af7cbc69cd247413. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.724576 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2ae19cc80194363b8a9c96fbafb4ebe" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:24.724824 23670 raft_consensus.cc:3060] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.727679 23670 raft_consensus.cc:2468] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.727707 23516 tablet_service.cc:1505] Processing CreateTablet for tablet de2ead20b8fd4053af7cbc69cd247413 (DEFAULT_TABLE table=default.Table_5a52905cde274ed386bff112ef98397a [id=1674ccf219e44d6e84ad9f62755ff1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.728229 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2ead20b8fd4053af7cbc69cd247413. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.733006 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:24.733601 23787 tablet_bootstrap.cc:492] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.733752 23787 ts_tablet_manager.cc:1403] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.155s user 0.014s sys 0.065s
I20251023 20:15:24.702260 23533 raft_consensus.cc:2468] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.734371 23604 leader_election.cc:304] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:24.734565 23787 raft_consensus.cc:359] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.734760 23787 raft_consensus.cc:385] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.734822 24198 raft_consensus.cc:2804] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.734861 23787 raft_consensus.cc:740] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.734977 24198 raft_consensus.cc:697] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:24.735116 23787 consensus_queue.cc:260] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.735231 24198 consensus_queue.cc:237] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.736711 23787 ts_tablet_manager.cc:1434] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:24.737833 23787 tablet_bootstrap.cc:492] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.740453 23787 tablet_bootstrap.cc:654] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.740880 24186 raft_consensus.cc:493] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.741132 24186 raft_consensus.cc:515] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.741873 24186 leader_election.cc:290] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.746518 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 68f5589f890f47e286a70419c2bf57c5 (DEFAULT_TABLE table=default.Table_5e2be9d05d7d431685748f79ce1c7311 [id=494f3980030c4d1fa22fec4ddcf14ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.746959 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f5589f890f47e286a70419c2bf57c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.750566 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 68f5589f890f47e286a70419c2bf57c5 (DEFAULT_TABLE table=default.Table_5e2be9d05d7d431685748f79ce1c7311 [id=494f3980030c4d1fa22fec4ddcf14ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.750978 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f5589f890f47e286a70419c2bf57c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.751816 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:24.758162 23371 tablet_service.cc:1505] Processing CreateTablet for tablet ea3149f906064f8c8167292ec86540fd (DEFAULT_TABLE table=default.Table_532d7e6d0eec4f09bcdf525c353fe0a8 [id=1c95889d5d3d408c9508b0d4509c84d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.758616 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3149f906064f8c8167292ec86540fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.767275 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 6313d3307aff4a6880176a06627e86fc (DEFAULT_TABLE table=default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 [id=3973b56caa79477f8542118421741084]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.767735 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6313d3307aff4a6880176a06627e86fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.768270 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 68f5589f890f47e286a70419c2bf57c5 (DEFAULT_TABLE table=default.Table_5e2be9d05d7d431685748f79ce1c7311 [id=494f3980030c4d1fa22fec4ddcf14ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.768664 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f5589f890f47e286a70419c2bf57c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.775779 23371 tablet_service.cc:1505] Processing CreateTablet for tablet de2ead20b8fd4053af7cbc69cd247413 (DEFAULT_TABLE table=default.Table_5a52905cde274ed386bff112ef98397a [id=1674ccf219e44d6e84ad9f62755ff1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.776196 24432 tablet_replica.cc:333] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:24.776270 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2ead20b8fd4053af7cbc69cd247413. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.776600 24432 raft_consensus.cc:2243] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:24.776985 24432 pending_rounds.cc:70] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:24.777129 23790 tablet_bootstrap.cc:492] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:24.777119 24432 pending_rounds.cc:77] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082146034425856 op_type: NO_OP noop_request { }
I20251023 20:15:24.777316 23790 ts_tablet_manager.cc:1403] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.192s user 0.014s sys 0.036s
I20251023 20:15:24.777402 24432 raft_consensus.cc:2889] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:24.777580 24432 raft_consensus.cc:2272] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:24.778201 23790 raft_consensus.cc:359] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.778393 23790 raft_consensus.cc:385] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.778591 23790 raft_consensus.cc:740] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.778831 23790 consensus_queue.cc:260] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.779312 23790 ts_tablet_manager.cc:1434] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:24.779639 23790 tablet_bootstrap.cc:492] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:24.779539 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 6313d3307aff4a6880176a06627e86fc (DEFAULT_TABLE table=default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 [id=3973b56caa79477f8542118421741084]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.780037 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6313d3307aff4a6880176a06627e86fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.780985 23786 tablet_bootstrap.cc:492] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.781162 23786 ts_tablet_manager.cc:1403] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.185s user 0.009s sys 0.039s
I20251023 20:15:24.782011 23786 raft_consensus.cc:359] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.782197 23786 raft_consensus.cc:385] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.782356 23786 raft_consensus.cc:740] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.782662 23786 consensus_queue.cc:260] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.783262 23786 ts_tablet_manager.cc:1434] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:24.783555 23786 tablet_bootstrap.cc:492] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.784268 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:24.785046 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.785286 23670 raft_consensus.cc:2468] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:24.785403 23532 raft_consensus.cc:1275] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Refusing update from remote peer d220aba7c9c0400398ffde917376ee48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:24.786005 24432 ts_tablet_manager.cc:1916] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:24.786405 23670 raft_consensus.cc:1275] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Refusing update from remote peer d220aba7c9c0400398ffde917376ee48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251023 20:15:24.789510 23786 tablet_bootstrap.cc:654] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.792062 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 38ce80a338d2494aac84fe6344d47e90 (DEFAULT_TABLE table=default.Table_5e48fffc45314e45a6f7ac8bba5ab31d [id=7260df2ce8034b73956131dd07502d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.788719 23333 leader_election.cc:304] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.789531 23790 tablet_bootstrap.cc:654] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.792529 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ce80a338d2494aac84fe6344d47e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.795120 24186 raft_consensus.cc:2804] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.795248 23333 leader_election.cc:304] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.795292 24186 raft_consensus.cc:697] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:24.795562 24186 consensus_queue.cc:237] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.796883 24198 raft_consensus.cc:493] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.797106 24220 raft_consensus.cc:2804] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.797199 24220 raft_consensus.cc:493] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.797295 24220 raft_consensus.cc:3060] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.797326 24198 raft_consensus.cc:515] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.798388 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244fa327fad74aad9e57a7fbcbf908bb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.798595 23398 raft_consensus.cc:2468] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.799046 23605 leader_election.cc:304] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.799377 24160 raft_consensus.cc:2804] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.799458 24160 raft_consensus.cc:493] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.799544 24160 raft_consensus.cc:3060] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.799945 24220 raft_consensus.cc:515] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.800714 24297 consensus_queue.cc:1048] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.802646 24198 leader_election.cc:290] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.800530 24220 leader_election.cc:290] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.805001 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244fa327fad74aad9e57a7fbcbf908bb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.806154 23604 leader_election.cc:343] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.807828 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 38ce80a338d2494aac84fe6344d47e90 (DEFAULT_TABLE table=default.Table_5e48fffc45314e45a6f7ac8bba5ab31d [id=7260df2ce8034b73956131dd07502d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.808254 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ce80a338d2494aac84fe6344d47e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.817589 24160 raft_consensus.cc:515] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.819221 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244fa327fad74aad9e57a7fbcbf908bb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:24.819450 23398 raft_consensus.cc:3060] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.822386 23398 raft_consensus.cc:2468] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.822937 23605 leader_election.cc:304] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:24.823395 24160 leader_election.cc:290] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.823588 24160 raft_consensus.cc:2804] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.823746 24160 raft_consensus.cc:697] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:24.824061 24160 consensus_queue.cc:237] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.826026 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244fa327fad74aad9e57a7fbcbf908bb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:24.827250 23604 leader_election.cc:343] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.828965 24198 raft_consensus.cc:493] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.829249 24198 raft_consensus.cc:515] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.830832 24198 leader_election.cc:290] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.831192 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95574277dde44983bb0a43d2b5ad01d5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.831362 23787 tablet_bootstrap.cc:492] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.831558 23787 ts_tablet_manager.cc:1403] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.094s user 0.029s sys 0.051s
W20251023 20:15:24.831847 23605 leader_election.cc:343] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.831956 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:24.832177 24428 raft_consensus.cc:493] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.832320 24341 consensus_queue.cc:1048] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.832406 23787 raft_consensus.cc:359] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.832532 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:24.832612 23787 raft_consensus.cc:385] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.832767 23787 raft_consensus.cc:740] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.832432 24428 raft_consensus.cc:515] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.832854 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
W20251023 20:15:24.832912 23333 leader_election.cc:343] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.833043 23787 consensus_queue.cc:260] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.833482 24341 consensus_queue.cc:1048] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.833614 23787 ts_tablet_manager.cc:1434] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:24.834219 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c317dc31cb0448d985512e6ba98735be" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:24.834283 24428 leader_election.cc:290] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.834424 23670 raft_consensus.cc:2468] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:24.834856 23470 leader_election.cc:304] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:24.835641 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:24.835820 23668 raft_consensus.cc:3060] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.835817 23787 tablet_bootstrap.cc:492] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.837324 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:24.837548 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:24.837754 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:24.838580 23787 tablet_bootstrap.cc:654] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.838881 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:24.838879 23668 raft_consensus.cc:2468] T 147781e046fe4feb9815147c774fe210 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:24.839212 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:24.839282 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 38ce80a338d2494aac84fe6344d47e90 (DEFAULT_TABLE table=default.Table_5e48fffc45314e45a6f7ac8bba5ab31d [id=7260df2ce8034b73956131dd07502d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.839385 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:24.839690 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ce80a338d2494aac84fe6344d47e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.839721 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:24.839876 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:24.840421 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.842110 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:24.842279 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:24.842401 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]). Expected version 1 got 0
I20251023 20:15:24.842526 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:24.842679 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:24.842849 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:24.842864 24383 raft_consensus.cc:2804] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.842978 24383 raft_consensus.cc:493] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.843075 24383 raft_consensus.cc:3060] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.843173 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:24.843357 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:24.843544 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:24.843847 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:24.844116 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:24.844393 23216 catalog_manager.cc:5649] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:24.844817 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]). Expected version 1 got 0
I20251023 20:15:24.847703 24383 raft_consensus.cc:515] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.848274 24383 leader_election.cc:290] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.850934 23787 tablet_bootstrap.cc:492] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:24.851115 23787 ts_tablet_manager.cc:1403] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20251023 20:15:24.851898 23787 raft_consensus.cc:359] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.852082 23787 raft_consensus.cc:385] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.852131 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:24.852170 23787 raft_consensus.cc:740] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.852411 23787 consensus_queue.cc:260] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.854002 23787 ts_tablet_manager.cc:1434] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:24.855127 23787 tablet_bootstrap.cc:492] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:24.855638 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c317dc31cb0448d985512e6ba98735be" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:24.855859 23668 raft_consensus.cc:3060] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.856626 24432 ts_tablet_manager.cc:1929] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:24.856758 24432 log.cc:1199] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/33b5efb30b5c491c8eafe3ac25688843
I20251023 20:15:24.856797 23787 tablet_bootstrap.cc:654] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.857614 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:24.857848 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:24.858049 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:24.858253 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
I20251023 20:15:24.858424 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:24.858599 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:24.858824 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_44c4bece35c24b7b9eae08ab8b835c78 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 1 got 0
I20251023 20:15:24.859639 23216 catalog_manager.cc:5649] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:24.860049 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]). Expected version 1 got 0
I20251023 20:15:24.860328 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:24.860926 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:24.861231 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]). Expected version 2 got 0
I20251023 20:15:24.861670 23668 raft_consensus.cc:2468] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:24.864495 23786 tablet_bootstrap.cc:492] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:24.864683 23786 ts_tablet_manager.cc:1403] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.004s
I20251023 20:15:24.865545 23786 raft_consensus.cc:359] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.865743 23786 raft_consensus.cc:385] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.865818 23786 raft_consensus.cc:740] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.866075 23786 consensus_queue.cc:260] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.866641 23786 ts_tablet_manager.cc:1434] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:24.861526 23216 catalog_manager.cc:5649] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:24.867336 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]). Expected version 2 got 0
I20251023 20:15:24.867779 23786 tablet_bootstrap.cc:492] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:24.867939 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:24.868263 23216 catalog_manager.cc:5649] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:24.868867 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]). Expected version 3 got 0
I20251023 20:15:24.869249 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]). Expected version 4 got 0
I20251023 20:15:24.869755 23216 catalog_manager.cc:5649] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:24.870201 23786 tablet_bootstrap.cc:654] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.870218 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]). Expected version 2 got 0
I20251023 20:15:24.870913 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:24.871353 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:24.871565 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971]). Expected version 1 got 0
I20251023 20:15:24.872184 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:24.872411 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:24.872620 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:24.872929 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]). Expected version 3 got 0
I20251023 20:15:24.873236 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:24.872859 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c317dc31cb0448d985512e6ba98735be" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.874591 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c317dc31cb0448d985512e6ba98735be" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:24.875348 23333 leader_election.cc:343] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.876657 23470 leader_election.cc:304] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:24.876953 24341 consensus_queue.cc:1048] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.877136 24259 raft_consensus.cc:2804] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.877468 24259 raft_consensus.cc:697] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:24.877467 24341 consensus_queue.cc:1048] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.877836 24259 consensus_queue.cc:237] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:24.878415 24383 raft_consensus.cc:493] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:24.878463 23210 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3150a8478c144d285e999d5c9270fe0 consensus state has no leader
I20251023 20:15:24.878542 24383 raft_consensus.cc:515] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.879169 24383 leader_election.cc:290] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.879082 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:24.879523 24341 consensus_queue.cc:1048] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:24.879714 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.880695 23333 leader_election.cc:304] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.881644 24341 raft_consensus.cc:2804] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:24.881798 24341 raft_consensus.cc:697] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
W20251023 20:15:24.881868 23470 leader_election.cc:343] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:24.882022 24341 consensus_queue.cc:237] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:24.884495 23333 consensus_peers.cc:597] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1a6fe726ef844829a7ffec9d4494637. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:24.884730 23333 consensus_peers.cc:597] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ea63540149c4c02abb8ae51209060a3. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:24.886067 23471 leader_election.cc:343] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:24.886670 23471 leader_election.cc:343] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.888731 24429 raft_consensus.cc:493] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.888891 24429 raft_consensus.cc:515] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.890178 23210 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_0febf0b2ad3345998c666d31603db1b1" } new_table_name: "default.Table_bbccd765c459437a838101139f1ec6c2" modify_external_catalogs: true
I20251023 20:15:24.890403 24429 leader_election.cc:290] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:24.890532 24432 ts_tablet_manager.cc:1950] T 33b5efb30b5c491c8eafe3ac25688843 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:24.892200 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4beebee06e5341a29c65d0c3cf985ef4" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.892477 23397 raft_consensus.cc:2393] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:24.895390 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8289ce37fa074891beb584bf2fff9747" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:24.901507 23211 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 118be7aa6380469db091d2788472e62b consensus state has no leader
I20251023 20:15:24.901990 23211 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_21f538f7569b437b98fe8a5274592880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:24.902669 23211 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21f538f7569b437b98fe8a5274592880 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.906936 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95574277dde44983bb0a43d2b5ad01d5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.907624 23604 leader_election.cc:343] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.907779 23604 leader_election.cc:304] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:24.908231 24198 raft_consensus.cc:2749] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:24.911501 23211 master_service.cc:954] ReplaceTablet: received request to replace tablet 58ae6a6e58274b8baf952eaaefde7127 from {username='slave'} at 127.0.0.1:55414
W20251023 20:15:24.912051 23204 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b3150a8478c144d285e999d5c9270fe0 consensus state has no leader
I20251023 20:15:24.914885 23217 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 1e5720c96730425dafb550094cf936cd
I20251023 20:15:24.915548 23217 master_service.cc:954] ReplaceTablet: received request to replace tablet 523d72012c364457b9a2a299f2f789a1 from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:24.921820 23471 leader_election.cc:304] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:24.922787 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:24.923141 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:24.924650 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
W20251023 20:15:24.924947 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
I20251023 20:15:24.924885 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8289ce37fa074891beb584bf2fff9747" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.925264 23397 raft_consensus.cc:2393] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:24.925444 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 33b5efb30b5c491c8eafe3ac25688843 (table default.Table_6d625b139835401587825737c53000e3 [id=2ede045acf474fe9a77d38eaed85a15a]) successfully deleted
I20251023 20:15:24.925522 24428 raft_consensus.cc:3060] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:24.925652 23205 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because the tablet was not found. No further retry: Not found: Tablet not found: 5acdc66df9c748a1a6300962f62fcf36
I20251023 20:15:24.928613 24428 raft_consensus.cc:2749] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:24.930325 23210 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59598ea40b9e4e5da0ec36f35f1094d6 consensus state has no leader
I20251023 20:15:24.931286 23210 master_service.cc:954] ReplaceTablet: received request to replace tablet f1a6fe726ef844829a7ffec9d4494637 from {username='slave'} at 127.0.0.1:55414
W20251023 20:15:24.931872 23215 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 255521d943d34713aaf773b15f9168c6 consensus state has no leader
I20251023 20:15:24.932143 23215 master_service.cc:954] ReplaceTablet: received request to replace tablet 994a555b24e4432a84601c54d422aa5d from {username='slave'} at 127.0.0.1:55414
W20251023 20:15:24.933015 23213 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea9be2a866cf46cfbec8db23a6d3f572 consensus state has no leader
I20251023 20:15:24.933389 23213 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_2496fe6d77e44734bfced35dee7d6596" } modify_external_catalogs: true
I20251023 20:15:24.933554 23213 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_2496fe6d77e44734bfced35dee7d6596" } modify_external_catalogs: true
I20251023 20:15:24.934669 23214 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 453eea5cdd2e47f29577c61b40ee59c0
I20251023 20:15:24.935364 23214 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_34d8a4d24be94b1ea676b7a0df81b96c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:24.935978 23214 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34d8a4d24be94b1ea676b7a0df81b96c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.937572 23471 leader_election.cc:304] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:24.937997 24184 raft_consensus.cc:3060] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.940546 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:24.940971 24184 raft_consensus.cc:2749] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:24.941782 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
I20251023 20:15:24.945559 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:24.946278 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
I20251023 20:15:24.948596 23214 master_service.cc:954] ReplaceTablet: received request to replace tablet 76380b341a304d6ba31648fbd1548f05 from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:24.950544 23212 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 39a62c97e0a3459e91c24c1c2f92389a
I20251023 20:15:24.951231 23212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_3d93ce2e43294364904450163de5e6d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:24.951841 23212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d93ce2e43294364904450163de5e6d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.955847 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:24.956873 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:24.958783 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
W20251023 20:15:24.959064 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
I20251023 20:15:24.963330 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:24.963418 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:24.963948 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:24.964852 23212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_3abcf10252074989b84e5566c762a99d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:24.965664 23212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3abcf10252074989b84e5566c762a99d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.970543 23790 tablet_bootstrap.cc:492] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:24.970770 23790 ts_tablet_manager.cc:1403] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.191s user 0.019s sys 0.020s
I20251023 20:15:24.971472 24341 consensus_queue.cc:1048] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.971612 23790 raft_consensus.cc:359] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.971953 23790 raft_consensus.cc:385] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.972199 23790 raft_consensus.cc:740] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.972453 24341 consensus_queue.cc:1048] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.972787 23790 consensus_queue.cc:260] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.973340 24341 consensus_queue.cc:1048] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:24.976202 23790 ts_tablet_manager.cc:1434] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:24.977615 23790 tablet_bootstrap.cc:492] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:24.979673 23790 tablet_bootstrap.cc:654] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:24.980278 24160 raft_consensus.cc:493] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:24.980726 24160 raft_consensus.cc:515] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.982296 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f47ddde9f3d4042bc84853332e21950" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:24.983443 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f47ddde9f3d4042bc84853332e21950" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:24.983655 23605 leader_election.cc:343] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.983690 23531 raft_consensus.cc:2468] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:24.983580 23212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_03ddad2769a4450b81d0d8028293a0b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:24.984093 23604 leader_election.cc:304] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
W20251023 20:15:24.984256 23212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03ddad2769a4450b81d0d8028293a0b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:24.984398 24198 raft_consensus.cc:2804] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:24.984508 24198 raft_consensus.cc:493] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:24.984591 24198 raft_consensus.cc:3060] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.987354 24198 raft_consensus.cc:515] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.988607 24198 leader_election.cc:290] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.989225 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f47ddde9f3d4042bc84853332e21950" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:24.989867 23605 leader_election.cc:343] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:24.990213 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f47ddde9f3d4042bc84853332e21950" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:24.991443 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 6313d3307aff4a6880176a06627e86fc (DEFAULT_TABLE table=default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 [id=3973b56caa79477f8542118421741084]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:24.991727 24160 leader_election.cc:290] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:24.992004 23534 raft_consensus.cc:3060] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:24.995546 23534 raft_consensus.cc:2468] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:24.992396 23790 tablet_bootstrap.cc:492] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:24.996562 23790 ts_tablet_manager.cc:1403] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20251023 20:15:24.996714 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6313d3307aff4a6880176a06627e86fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:24.997551 23790 raft_consensus.cc:359] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.997844 23790 raft_consensus.cc:385] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:24.997983 23790 raft_consensus.cc:740] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:24.998272 23790 consensus_queue.cc:260] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:24.999698 23790 ts_tablet_manager.cc:1434] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251023 20:15:24.999890 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
W20251023 20:15:25.000495 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
W20251023 20:15:25.001073 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
I20251023 20:15:25.001335 23790 tablet_bootstrap.cc:492] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.003728 23790 tablet_bootstrap.cc:654] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.005604 23212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_94f35966189741de98430b366fccb99f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:25.007411 23212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94f35966189741de98430b366fccb99f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:25.021991 23790 tablet_bootstrap.cc:492] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.022264 23790 ts_tablet_manager.cc:1403] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20251023 20:15:25.023110 23790 raft_consensus.cc:359] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.023420 23790 raft_consensus.cc:385] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.023536 23790 raft_consensus.cc:740] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.023871 23790 consensus_queue.cc:260] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.025481 23790 ts_tablet_manager.cc:1434] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:25.026479 23790 tablet_bootstrap.cc:492] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.028066 23790 tablet_bootstrap.cc:654] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:25.029197 23203 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.189:33509, user_credentials={real_user=slave}} blocked reactor thread for 53964us
I20251023 20:15:25.034240 23217 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_95fb2c2528f44d8d84f4cd67d9663502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:25.034919 23217 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95fb2c2528f44d8d84f4cd67d9663502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:25.038487 23333 consensus_peers.cc:597] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef1229b9e0384a82811f244d2bf4c8ce. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.038882 24186 consensus_queue.cc:1048] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.041239 23790 tablet_bootstrap.cc:492] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.041455 23790 ts_tablet_manager.cc:1403] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20251023 20:15:25.042354 23790 raft_consensus.cc:359] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.042623 23790 raft_consensus.cc:385] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.042754 23790 raft_consensus.cc:740] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.043069 23790 consensus_queue.cc:260] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.044477 23790 ts_tablet_manager.cc:1434] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:25.045547 23790 tablet_bootstrap.cc:492] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.047394 23790 tablet_bootstrap.cc:654] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.047540 23213 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 3befdfc85bca42bb816e11d1dd832c8c
I20251023 20:15:25.048270 23213 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_c37d72052930485b819533573f3710ac" } modify_external_catalogs: true
I20251023 20:15:25.048466 23213 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_c37d72052930485b819533573f3710ac" } modify_external_catalogs: true
I20251023 20:15:25.053251 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.054064 23333 consensus_peers.cc:597] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7301d6121c9d4c8dbc642e895aa4e9f6. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.054854 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
W20251023 20:15:25.059610 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
I20251023 20:15:25.069850 23787 tablet_bootstrap.cc:492] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.070084 23787 ts_tablet_manager.cc:1403] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.215s user 0.074s sys 0.128s
I20251023 20:15:25.070919 23787 raft_consensus.cc:359] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.071159 23787 raft_consensus.cc:385] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.071262 23787 raft_consensus.cc:740] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.071565 23787 consensus_queue.cc:260] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.072098 23787 ts_tablet_manager.cc:1434] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:25.013609 24184 raft_consensus.cc:493] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.073189 23787 tablet_bootstrap.cc:492] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.016404 23786 tablet_bootstrap.cc:492] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:25.073800 23604 leader_election.cc:304] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:25.074287 24160 raft_consensus.cc:2804] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.076028 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:25.077258 23214 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 76380b341a304d6ba31648fbd1548f05
I20251023 20:15:25.077736 23211 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 58ae6a6e58274b8baf952eaaefde7127
I20251023 20:15:25.077736 23214 catalog_manager.cc:6719] ReplaceTablet: tablet 76380b341a304d6ba31648fbd1548f05 deleted and replaced by tablet e289b51f48a942fe91b95f528130f89b
I20251023 20:15:25.078142 23211 catalog_manager.cc:6719] ReplaceTablet: tablet 58ae6a6e58274b8baf952eaaefde7127 deleted and replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3
I20251023 20:15:25.079774 24462 tablet.cc:1722] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:25.084456 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:25.084944 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
I20251023 20:15:25.085042 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:25.085897 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
W20251023 20:15:25.086081 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 58ae6a6e58274b8baf952eaaefde7127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58ae6a6e58274b8baf952eaaefde7127 already in progress: creating tablet
I20251023 20:15:25.091321 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.091639 24160 raft_consensus.cc:697] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:25.092263 24160 consensus_queue.cc:237] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:25.092518 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
I20251023 20:15:25.092506 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.093036 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 58ae6a6e58274b8baf952eaaefde7127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58ae6a6e58274b8baf952eaaefde7127 already in progress: creating tablet
I20251023 20:15:25.094542 24465 tablet.cc:1722] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:25.105482 23787 tablet_bootstrap.cc:654] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.106642 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.106822 23786 ts_tablet_manager.cc:1403] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.206s user 0.011s sys 0.034s
I20251023 20:15:25.107690 24184 raft_consensus.cc:515] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.108254 23790 tablet_bootstrap.cc:492] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.109212 23790 ts_tablet_manager.cc:1403] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.018s
I20251023 20:15:25.110451 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:25.110651 23790 raft_consensus.cc:359] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.110860 23790 raft_consensus.cc:385] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.110921 23790 raft_consensus.cc:740] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.111177 23790 consensus_queue.cc:260] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.112380 23790 ts_tablet_manager.cc:1434] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:25.113299 23790 tablet_bootstrap.cc:492] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.113945 23786 raft_consensus.cc:359] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.114128 23786 raft_consensus.cc:385] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.114085 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "523d72012c364457b9a2a299f2f789a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.114091 24198 raft_consensus.cc:493] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.114291 23669 raft_consensus.cc:2468] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:25.114315 24198 raft_consensus.cc:515] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.114926 24198 leader_election.cc:290] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.115310 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a4c8dcbdfcf4a15b8a4194ea04b93a0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.115803 23605 leader_election.cc:343] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.116297 23786 raft_consensus.cc:740] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.116537 23786 consensus_queue.cc:260] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.119037 23790 tablet_bootstrap.cc:654] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.121162 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.121765 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.123076 23787 tablet_bootstrap.cc:492] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.123227 23787 ts_tablet_manager.cc:1403] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.003s
I20251023 20:15:25.123401 24184 leader_election.cc:290] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.123818 23470 leader_election.cc:304] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:25.123975 23787 raft_consensus.cc:359] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.124158 23787 raft_consensus.cc:385] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.124238 23787 raft_consensus.cc:740] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.124439 23787 consensus_queue.cc:260] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.124663 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a4c8dcbdfcf4a15b8a4194ea04b93a0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:25.125507 23604 leader_election.cc:343] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.125641 23604 leader_election.cc:304] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.126446 24428 consensus_queue.cc:1048] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.129280 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "523d72012c364457b9a2a299f2f789a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.129470 23397 raft_consensus.cc:2468] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:25.130041 24198 raft_consensus.cc:2749] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.130033 24341 consensus_queue.cc:1048] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.130254 24186 consensus_queue.cc:1048] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.130550 23787 ts_tablet_manager.cc:1434] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:25.130877 23787 tablet_bootstrap.cc:492] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.132381 23790 tablet_bootstrap.cc:492] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.132536 23787 tablet_bootstrap.cc:654] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.132645 23790 ts_tablet_manager.cc:1403] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20251023 20:15:25.139741 24184 raft_consensus.cc:2804] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.140728 24184 raft_consensus.cc:493] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.140869 24184 raft_consensus.cc:3060] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.142957 24198 raft_consensus.cc:493] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.143126 24198 raft_consensus.cc:515] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.143801 24198 leader_election.cc:290] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.143863 23786 ts_tablet_manager.cc:1434] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20251023 20:15:25.144152 23786 tablet_bootstrap.cc:492] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:25.145463 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532c8567db14901b383f42e5f04b4c2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.145758 23787 tablet_bootstrap.cc:492] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.145915 23787 ts_tablet_manager.cc:1403] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
W20251023 20:15:25.146217 23604 leader_election.cc:343] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.146605 23787 raft_consensus.cc:359] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.146802 23787 raft_consensus.cc:385] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.146865 23787 raft_consensus.cc:740] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.147115 23787 consensus_queue.cc:260] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.148247 23787 ts_tablet_manager.cc:1434] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:25.148295 24184 raft_consensus.cc:515] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.148903 23787 tablet_bootstrap.cc:492] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.149703 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "523d72012c364457b9a2a299f2f789a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:25.149930 24184 leader_election.cc:290] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.149930 23669 raft_consensus.cc:3060] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.150780 23787 tablet_bootstrap.cc:654] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.150946 24220 consensus_queue.cc:1048] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.151510 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532c8567db14901b383f42e5f04b4c2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.152065 23605 leader_election.cc:343] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.152184 23605 leader_election.cc:304] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.152415 24198 raft_consensus.cc:2749] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.152743 23669 raft_consensus.cc:2468] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:25.153432 23470 leader_election.cc:304] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:25.153784 24184 raft_consensus.cc:2804] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.153954 24184 raft_consensus.cc:697] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:25.154311 24184 consensus_queue.cc:237] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.154907 23790 raft_consensus.cc:359] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:25.155085 23790 raft_consensus.cc:385] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.155174 23790 raft_consensus.cc:740] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.155377 23790 consensus_queue.cc:260] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:25.155957 23790 ts_tablet_manager.cc:1434] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.023s user 0.000s sys 0.000s
I20251023 20:15:25.156252 23790 tablet_bootstrap.cc:492] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:25.161558 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
W20251023 20:15:25.161840 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
W20251023 20:15:25.162043 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 58ae6a6e58274b8baf952eaaefde7127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58ae6a6e58274b8baf952eaaefde7127 already in progress: creating tablet
I20251023 20:15:25.162828 23786 tablet_bootstrap.cc:654] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.162982 23787 tablet_bootstrap.cc:492] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.163136 23787 ts_tablet_manager.cc:1403] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20251023 20:15:25.163951 23787 raft_consensus.cc:359] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.164136 23787 raft_consensus.cc:385] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.164237 23787 raft_consensus.cc:740] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.164516 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "523d72012c364457b9a2a299f2f789a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:25.165009 23397 raft_consensus.cc:3060] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.165661 23787 consensus_queue.cc:260] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.167371 23787 ts_tablet_manager.cc:1434] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:25.168685 23787 tablet_bootstrap.cc:492] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.171767 23787 tablet_bootstrap.cc:654] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.183101 23397 raft_consensus.cc:2468] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:25.183490 23333 consensus_peers.cc:597] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fbad98d3ab24cc5b7726a92fba04cf8. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.183183 23787 tablet_bootstrap.cc:492] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.186957 23790 tablet_bootstrap.cc:654] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:25.192234 23470 consensus_peers.cc:597] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83da09ee0d10435ba89a19dc378d34eb. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.204294 24198 raft_consensus.cc:493] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.204479 24198 raft_consensus.cc:515] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.205261 24198 leader_election.cc:290] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.206661 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5964da70989b4e6cb77fe5d0a52ce9cf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.207249 23605 leader_election.cc:343] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.210777 24220 consensus_queue.cc:1048] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.215338 24198 consensus_queue.cc:1048] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.225785 23210 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 22) took 1113 ms. Trace:
I20251023 20:15:25.225960 23210 rpcz_store.cc:276] 1023 20:15:24.112668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:24.931184 (+818516us) service_pool.cc:225] Handling call
1023 20:15:25.225720 (+294536us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:25.230531 24174 raft_consensus.cc:493] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.230686 24174 raft_consensus.cc:515] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.231262 24174 leader_election.cc:290] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.232255 23790 tablet_bootstrap.cc:492] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.232564 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.233228 23605 leader_election.cc:343] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.234066 23790 ts_tablet_manager.cc:1403] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.006s
I20251023 20:15:25.235477 24184 raft_consensus.cc:493] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.237705 23790 raft_consensus.cc:359] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.237948 23790 raft_consensus.cc:385] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.238021 23790 raft_consensus.cc:740] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.238257 23790 consensus_queue.cc:260] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.239073 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5964da70989b4e6cb77fe5d0a52ce9cf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:25.239976 23604 leader_election.cc:343] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.240125 23604 leader_election.cc:304] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.240577 24174 raft_consensus.cc:2749] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.243649 24341 consensus_queue.cc:1048] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.246038 23790 ts_tablet_manager.cc:1434] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251023 20:15:25.246368 23790 tablet_bootstrap.cc:492] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.248107 23790 tablet_bootstrap.cc:654] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.184278 23787 ts_tablet_manager.cc:1403] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.001s
I20251023 20:15:25.251772 23787 raft_consensus.cc:359] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.253530 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:25.253909 23470 consensus_peers.cc:597] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cff337a6e728489b99fa8c5d4dfece2b. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:25.254460 23604 leader_election.cc:343] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.254993 23604 leader_election.cc:304] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.255120 23536 raft_consensus.cc:3060] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.255450 24174 raft_consensus.cc:2749] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.255805 23787 raft_consensus.cc:385] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.256292 23787 raft_consensus.cc:740] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.257009 23787 consensus_queue.cc:260] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.259918 24174 consensus_queue.cc:1048] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.263628 23787 ts_tablet_manager.cc:1434] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251023 20:15:25.261638 24184 raft_consensus.cc:515] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.264876 23530 raft_consensus.cc:3060] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.265424 24184 leader_election.cc:290] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.265548 23790 tablet_bootstrap.cc:492] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.265712 23790 ts_tablet_manager.cc:1403] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20251023 20:15:25.266537 23790 raft_consensus.cc:359] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.266697 23790 raft_consensus.cc:385] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.266767 23790 raft_consensus.cc:740] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.266969 23790 consensus_queue.cc:260] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.267426 23790 ts_tablet_manager.cc:1434] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:25.267696 23790 tablet_bootstrap.cc:492] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.268355 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a2f3adb5d347a0adc198cd1ef93304" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.270140 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a2f3adb5d347a0adc198cd1ef93304" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.270658 24174 consensus_queue.cc:1048] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.271539 23470 leader_election.cc:304] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:25.274014 23787 tablet_bootstrap.cc:492] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.276247 23787 tablet_bootstrap.cc:654] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.292659 24495 raft_consensus.cc:493] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.292929 24472 tablet.cc:1722] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:25.292922 24495 raft_consensus.cc:515] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:25.293931 24495 leader_election.cc:290] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.294945 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c317dc31cb0448d985512e6ba98735be" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.296226 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c317dc31cb0448d985512e6ba98735be" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.296473 23668 raft_consensus.cc:2393] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:25.296999 23333 leader_election.cc:304] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.297544 24495 raft_consensus.cc:3060] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.300194 23217 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_31442b53b2f748d0895b0014aa8ae96f" } new_table_name: "default.Table_8322279a1b514373a3113f54d917fb55" modify_external_catalogs: true
I20251023 20:15:25.300755 23790 tablet_bootstrap.cc:654] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.302217 23787 tablet_bootstrap.cc:492] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.302371 23787 ts_tablet_manager.cc:1403] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20251023 20:15:25.303156 23787 raft_consensus.cc:359] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.303325 23787 raft_consensus.cc:385] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.303411 23787 raft_consensus.cc:740] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.304173 23214 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_222ab66f7150462aad2efe8d16484b94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:25.304742 23214 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_222ab66f7150462aad2efe8d16484b94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:25.306128 23211 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_2afbf83f519f4b7da2c4acc31ab18380"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:25.306577 23211 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2afbf83f519f4b7da2c4acc31ab18380 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:25.306815 24495 raft_consensus.cc:2749] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.308355 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_64ad686282f94cb7ac869f59933669f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:25.308827 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64ad686282f94cb7ac869f59933669f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:25.308895 23790 tablet_bootstrap.cc:492] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.309039 23790 ts_tablet_manager.cc:1403] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20251023 20:15:25.309756 23790 raft_consensus.cc:359] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.309940 23790 raft_consensus.cc:385] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.310012 23790 raft_consensus.cc:740] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.310220 23790 consensus_queue.cc:260] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.310738 23790 ts_tablet_manager.cc:1434] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:25.311033 23790 tablet_bootstrap.cc:492] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.312739 23787 consensus_queue.cc:260] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.314146 23787 ts_tablet_manager.cc:1434] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251023 20:15:25.314908 23787 tablet_bootstrap.cc:492] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.316730 23787 tablet_bootstrap.cc:654] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.320303 23790 tablet_bootstrap.cc:654] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.322381 23786 tablet_bootstrap.cc:492] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:25.322558 23786 ts_tablet_manager.cc:1403] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.178s user 0.008s sys 0.002s
I20251023 20:15:25.323302 23786 raft_consensus.cc:359] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.323504 23786 raft_consensus.cc:385] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.323583 23786 raft_consensus.cc:740] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:25.324851 23333 consensus_peers.cc:597] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52bc660b9da54c5d984381a267e1efa2. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.325728 24259 raft_consensus.cc:2749] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.331704 23790 tablet_bootstrap.cc:492] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.332101 23790 ts_tablet_manager.cc:1403] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20251023 20:15:25.332958 23790 raft_consensus.cc:359] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.333173 23790 raft_consensus.cc:385] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.333264 23790 raft_consensus.cc:740] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.333493 23790 consensus_queue.cc:260] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.334029 23790 ts_tablet_manager.cc:1434] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:25.334338 23790 tablet_bootstrap.cc:492] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.336165 23790 tablet_bootstrap.cc:654] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.338572 23787 tablet_bootstrap.cc:492] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.339668 23787 ts_tablet_manager.cc:1403] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.001s
I20251023 20:15:25.342092 24495 consensus_queue.cc:1048] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.351450 23787 raft_consensus.cc:359] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.353205 23787 raft_consensus.cc:385] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.353869 23787 raft_consensus.cc:740] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.354996 23787 consensus_queue.cc:260] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.355381 23215 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 994a555b24e4432a84601c54d422aa5d
I20251023 20:15:25.355958 23215 catalog_manager.cc:6719] ReplaceTablet: tablet 994a555b24e4432a84601c54d422aa5d deleted and replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac
I20251023 20:15:25.356065 23215 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 23) took 1243 ms. Trace:
I20251023 20:15:25.356741 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.356166 23215 rpcz_store.cc:276] 1023 20:15:24.112751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:24.932067 (+819316us) service_pool.cc:225] Handling call
1023 20:15:25.356038 (+423971us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:25.301968 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:25.302078 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:25.302085 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:25.302088 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:25.302216 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:25.302226 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:25.302231 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:25.302306 (+ 75us) tablet.cc:699] Row locks acquired
1023 20:15:25.302308 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:25.302341 (+ 33us) write_op.cc:270] Start()
1023 20:15:25.302375 (+ 34us) write_op.cc:276] Timestamp: P: 1761250525302300 usec, L: 0
1023 20:15:25.302379 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:25.302522 (+ 143us) log.cc:844] Serialized 631 byte log entry
1023 20:15:25.352482 (+ 49960us) op_driver.cc:464] REPLICATION: finished
1023 20:15:25.354071 (+ 1589us) write_op.cc:301] APPLY: starting
1023 20:15:25.354085 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:25.354121 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:25.354146 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:25.354241 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:25.354772 (+ 531us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:25.354780 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:25.354853 (+ 73us) log.cc:844] Serialized 29 byte log entry
1023 20:15:25.354927 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.354945 (+ 18us) write_op.cc:454] Released schema lock
1023 20:15:25.354951 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:25.355014 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.355018 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1537,"apply.run_cpu_time_us":547,"apply.run_wall_time_us":945,"num_ops":2,"prepare.queue_time_us":202244,"prepare.run_cpu_time_us":641,"prepare.run_wall_time_us":640,"raft.queue_time_us":3226,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":50087,"spinlock_wait_cycles":34560}]]}
I20251023 20:15:25.357136 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:25.357437 23787 ts_tablet_manager.cc:1434] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251023 20:15:25.357757 23787 tablet_bootstrap.cc:492] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:25.357867 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 994a555b24e4432a84601c54d422aa5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994a555b24e4432a84601c54d422aa5d already in progress: creating tablet
I20251023 20:15:25.358214 23213 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet c1f9935f5e4f4daa9df8fe1611ed76c9
W20251023 20:15:25.358354 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 994a555b24e4432a84601c54d422aa5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994a555b24e4432a84601c54d422aa5d already in progress: creating tablet
I20251023 20:15:25.358953 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.359303 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:25.359360 23787 tablet_bootstrap.cc:654] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:25.359885 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
I20251023 20:15:25.359916 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.359952 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.360297 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
I20251023 20:15:25.361671 23790 tablet_bootstrap.cc:492] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.361845 23790 ts_tablet_manager.cc:1403] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
W20251023 20:15:25.362350 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
W20251023 20:15:25.362591 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 994a555b24e4432a84601c54d422aa5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994a555b24e4432a84601c54d422aa5d already in progress: creating tablet
I20251023 20:15:25.362702 23790 raft_consensus.cc:359] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.362910 23790 raft_consensus.cc:385] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.362988 23790 raft_consensus.cc:740] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.363242 23790 consensus_queue.cc:260] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.364442 23790 ts_tablet_manager.cc:1434] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:15:25.366107 23790 tablet_bootstrap.cc:492] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.368081 23790 tablet_bootstrap.cc:654] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.382469 23790 tablet_bootstrap.cc:492] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.382752 23790 ts_tablet_manager.cc:1403] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
W20251023 20:15:25.382962 23333 consensus_peers.cc:597] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.383638 23790 raft_consensus.cc:359] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.383831 23790 raft_consensus.cc:385] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.383920 23790 raft_consensus.cc:740] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.384176 23790 consensus_queue.cc:260] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.385419 23790 ts_tablet_manager.cc:1434] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:25.386068 23790 tablet_bootstrap.cc:492] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.388264 23790 tablet_bootstrap.cc:654] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.392572 23787 tablet_bootstrap.cc:492] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.392733 23787 ts_tablet_manager.cc:1403] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20251023 20:15:25.396454 23787 raft_consensus.cc:359] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.396651 23787 raft_consensus.cc:385] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.396742 23787 raft_consensus.cc:740] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.398168 24198 raft_consensus.cc:493] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.398310 24198 raft_consensus.cc:515] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.398536 24174 consensus_queue.cc:1048] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.398921 24198 leader_election.cc:290] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.400667 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5346a221c78342359eb6e418bd12cd24" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.401888 23605 leader_election.cc:343] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.402587 23790 tablet_bootstrap.cc:492] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.402792 23790 ts_tablet_manager.cc:1403] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20251023 20:15:25.402889 24383 consensus_queue.cc:1048] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.403652 23790 raft_consensus.cc:359] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.403898 23790 raft_consensus.cc:385] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.403997 23790 raft_consensus.cc:740] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.404274 23790 consensus_queue.cc:260] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.410823 23786 consensus_queue.cc:260] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.411471 23786 ts_tablet_manager.cc:1434] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20251023 20:15:25.414251 24174 consensus_queue.cc:1048] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.418881 23212 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_3d93ce2e43294364904450163de5e6d7" } new_table_name: "default.Table_6621b13baee14821a56e8610ed7c68a7" modify_external_catalogs: true
I20251023 20:15:25.418841 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_ef7d0ab458b247c5a1998f0538a7b2ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:25.419485 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef7d0ab458b247c5a1998f0538a7b2ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:25.420717 23213 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_1ec08f875c4f4a438cd2081db4326eb3" } modify_external_catalogs: true
I20251023 20:15:25.420897 23213 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_1ec08f875c4f4a438cd2081db4326eb3" } modify_external_catalogs: true
I20251023 20:15:25.423595 23790 ts_tablet_manager.cc:1434] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.021s user 0.000s sys 0.001s
I20251023 20:15:25.424427 23790 tablet_bootstrap.cc:492] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:25.427610 23604 consensus_peers.cc:597] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:25.430858 23604 consensus_peers.cc:597] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.434065 24230 raft_consensus.cc:493] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.434239 24230 raft_consensus.cc:515] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.438062 24198 raft_consensus.cc:493] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.438230 24198 raft_consensus.cc:515] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.438812 24198 leader_election.cc:290] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:25.439513 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:25.440119 23604 leader_election.cc:343] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.441282 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.444972 23605 leader_election.cc:343] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.445147 23605 leader_election.cc:304] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.447002 23790 tablet_bootstrap.cc:654] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.448243 24198 raft_consensus.cc:2749] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.452572 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:25.455107 23787 consensus_queue.cc:260] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.457162 24230 leader_election.cc:290] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.457641 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5346a221c78342359eb6e418bd12cd24" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.458542 24198 raft_consensus.cc:493] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.458700 24198 raft_consensus.cc:515] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.459275 24198 leader_election.cc:290] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.460419 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa794b3bec4b4b3993c814ea7cd02ee2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.460772 23398 raft_consensus.cc:2468] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:25.461148 23605 leader_election.cc:304] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:25.461408 24198 raft_consensus.cc:2804] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.461495 24198 raft_consensus.cc:493] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.461612 24198 raft_consensus.cc:3060] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:25.463182 23217 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0eb7935c4cf64e2f802ca12c1f63317b consensus state has no leader
I20251023 20:15:25.464293 24198 raft_consensus.cc:515] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.465790 24198 leader_election.cc:290] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.467612 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5964da70989b4e6cb77fe5d0a52ce9cf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.467772 23668 raft_consensus.cc:2468] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:25.468111 23333 leader_election.cc:304] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:25.468367 24339 raft_consensus.cc:493] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.468482 24339 raft_consensus.cc:515] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.469049 24339 leader_election.cc:290] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.469213 24339 raft_consensus.cc:2804] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.469334 24339 raft_consensus.cc:493] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.469427 24339 raft_consensus.cc:3060] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.472103 24339 raft_consensus.cc:515] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.472630 24339 leader_election.cc:290] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.473630 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5964da70989b4e6cb77fe5d0a52ce9cf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:25.473825 23667 raft_consensus.cc:3060] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.476471 23667 raft_consensus.cc:2468] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:25.476958 23333 leader_election.cc:304] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:25.477348 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a21ec49194e24723b31fd9c86f1f85e4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.477600 23670 raft_consensus.cc:2468] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:25.478053 23333 leader_election.cc:304] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:25.478430 24339 raft_consensus.cc:2804] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.478699 24495 raft_consensus.cc:2804] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.478791 24495 raft_consensus.cc:493] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.478890 24495 raft_consensus.cc:3060] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.481703 24495 raft_consensus.cc:515] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.482318 24495 leader_election.cc:290] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.483037 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a21ec49194e24723b31fd9c86f1f85e4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:25.483258 23670 raft_consensus.cc:3060] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.487655 24339 raft_consensus.cc:697] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:25.488030 24339 consensus_queue.cc:237] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.491338 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa794b3bec4b4b3993c814ea7cd02ee2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:25.494630 23787 ts_tablet_manager.cc:1434] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.102s user 0.002s sys 0.000s
I20251023 20:15:25.495638 23398 raft_consensus.cc:3060] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.499590 24198 raft_consensus.cc:493] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.499883 24198 raft_consensus.cc:515] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.500968 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8ba7ab28dfa4ac38e0b58ba2115495c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.501477 23605 leader_election.cc:343] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.502454 24198 leader_election.cc:290] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.459887 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa794b3bec4b4b3993c814ea7cd02ee2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.476233 24383 raft_consensus.cc:493] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.502808 24383 raft_consensus.cc:515] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:25.503178 23604 leader_election.cc:343] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.503389 24383 leader_election.cc:290] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.503954 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edec01a63a3046d9ae5a372fc9068b52" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.491890 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa794b3bec4b4b3993c814ea7cd02ee2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:25.504036 23604 leader_election.cc:343] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.504284 23669 raft_consensus.cc:2393] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:25.504237 23604 leader_election.cc:304] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.505133 24174 raft_consensus.cc:2749] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:25.505203 23604 leader_election.cc:343] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.505318 24174 raft_consensus.cc:493] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.501813 23786 tablet_bootstrap.cc:492] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:25.505439 24174 raft_consensus.cc:515] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.505980 24174 leader_election.cc:290] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:25.502310 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8ba7ab28dfa4ac38e0b58ba2115495c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:25.506935 23604 leader_election.cc:343] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.507048 23604 leader_election.cc:304] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.507330 23786 tablet_bootstrap.cc:654] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.507385 24174 raft_consensus.cc:2749] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.507921 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d82c92e863b2470699cc822b8cb8c14d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:25.508432 23604 leader_election.cc:343] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.508912 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d82c92e863b2470699cc822b8cb8c14d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.509341 23787 tablet_bootstrap.cc:492] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:25.510032 23605 leader_election.cc:343] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.511966 23605 leader_election.cc:304] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.512311 24174 raft_consensus.cc:2749] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.514237 23787 tablet_bootstrap.cc:654] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.518201 23790 tablet_bootstrap.cc:492] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:25.518159 23471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 92633.9us
I20251023 20:15:25.518353 23790 ts_tablet_manager.cc:1403] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.095s user 0.026s sys 0.009s
I20251023 20:15:25.518798 23211 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_48c1e728bef74781a6d7353e1b1181be" } new_table_name: "default.Table_40171fa2448745aaa4c50eac538f6e85" modify_external_catalogs: true
I20251023 20:15:25.517892 23670 raft_consensus.cc:2468] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:25.520351 23790 raft_consensus.cc:359] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.520534 23790 raft_consensus.cc:385] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.520635 23790 raft_consensus.cc:740] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.520879 23790 consensus_queue.cc:260] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.520876 23333 leader_election.cc:304] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:25.521378 23790 ts_tablet_manager.cc:1434] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:25.521670 23790 tablet_bootstrap.cc:492] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.522536 23398 raft_consensus.cc:2468] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:25.523228 24174 consensus_queue.cc:1048] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.525637 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5964da70989b4e6cb77fe5d0a52ce9cf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.528084 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5964da70989b4e6cb77fe5d0a52ce9cf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:25.528622 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a21ec49194e24723b31fd9c86f1f85e4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:25.529448 23787 tablet_bootstrap.cc:492] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.529633 23787 ts_tablet_manager.cc:1403] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20251023 20:15:25.529660 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a21ec49194e24723b31fd9c86f1f85e4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.530438 23787 raft_consensus.cc:359] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.530643 23787 raft_consensus.cc:385] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.530721 23787 raft_consensus.cc:740] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.530972 23787 consensus_queue.cc:260] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.534039 23787 ts_tablet_manager.cc:1434] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:25.534385 23787 tablet_bootstrap.cc:492] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.536156 23787 tablet_bootstrap.cc:654] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.536993 23210 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_e6177be8581a457cb9113fcc941d6c65" } new_table_name: "default.Table_7eb5076c09144543885abb4392070ad8" modify_external_catalogs: true
I20251023 20:15:25.523291 23790 tablet_bootstrap.cc:654] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.524875 24495 raft_consensus.cc:2804] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.541622 24495 raft_consensus.cc:697] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:25.541942 24495 consensus_queue.cc:237] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:25.529166 23333 leader_election.cc:343] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:25.543241 23333 leader_election.cc:343] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:25.543540 23333 leader_election.cc:343] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:25.543859 23333 leader_election.cc:343] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.530310 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "edec01a63a3046d9ae5a372fc9068b52" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.544982 23471 leader_election.cc:304] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:25.545420 24514 raft_consensus.cc:3060] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.548275 24514 raft_consensus.cc:2749] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.548971 24174 consensus_queue.cc:1048] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.549374 23605 leader_election.cc:304] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.549906 23786 tablet_bootstrap.cc:492] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:25.550266 24174 raft_consensus.cc:2804] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.550410 24174 raft_consensus.cc:697] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:25.550669 24174 consensus_queue.cc:237] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:25.552831 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a744214a24cc4bf28654ffa417ce0d3d consensus state has no leader
W20251023 20:15:25.553009 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e5af107f9a144edb9080b7135ab8aeb9 consensus state has no leader
W20251023 20:15:25.553210 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9b766ff2dff34a8f80f70591dba54017 consensus state has no leader
I20251023 20:15:25.553300 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.553383 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8a630b83f614447ebf2a08851d6b827d consensus state has no leader
I20251023 20:15:25.553566 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.553823 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 757d41e027ba461ea051b38a2d685407 consensus state has no leader
W20251023 20:15:25.554061 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet eb5a6d9a4509431f97b69a3b138b5829 consensus state has no leader
W20251023 20:15:25.554376 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 26a680ad3aed487593bd1b63004dbbfc consensus state has no leader
W20251023 20:15:25.555714 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 124703edbf08408a9ca69732674c0ad3 consensus state has no leader
I20251023 20:15:25.556717 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.556936 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.557121 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.557294 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.557461 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.557632 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.557633 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet af899c01b4af46d39d27f984cc3d5940 consensus state has no leader
I20251023 20:15:25.557802 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.557897 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a04c65155ce048cb82b150cbe20be849 consensus state has no leader
I20251023 20:15:25.557977 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.558547 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0ca01bea2bb146ca8797a9592a840a81 consensus state has no leader
I20251023 20:15:25.558586 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.558784 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.558938 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 04f7dffd94f3442ea20045a512d7bd65 consensus state has no leader
I20251023 20:15:25.559247 24363 tablet_replica.cc:333] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:25.559224 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet b680b1cbc103403b91a1104248ecbdcf consensus state has no leader
I20251023 20:15:25.559432 24363 raft_consensus.cc:2243] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:25.559458 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet efad9b4a8e2e446f917df7ca26fbf789 consensus state has no leader
I20251023 20:15:25.559672 24363 pending_rounds.cc:70] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Trying to abort 1 pending ops.
I20251023 20:15:25.559726 24363 pending_rounds.cc:77] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082152657997824 op_type: NO_OP noop_request { }
I20251023 20:15:25.559862 24363 raft_consensus.cc:2889] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:25.559966 24363 raft_consensus.cc:2272] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:25.560835 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 82ae1a94b011412c851cd96946744fc4 consensus state has no leader
W20251023 20:15:25.561275 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 744d31fccbbe4fa59c56e24136e3b2b9 consensus state has no leader
W20251023 20:15:25.561645 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 2fea0afc8f63497d8b5cdf2352a20267 consensus state has no leader
W20251023 20:15:25.562546 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a570f339890946b7ab20c15de68462d5 consensus state has no leader
W20251023 20:15:25.565287 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 53f33020319d48c791e1156c67b55be6 consensus state has no leader
W20251023 20:15:25.568562 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
W20251023 20:15:25.568769 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
I20251023 20:15:25.570272 24160 consensus_queue.cc:1048] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:25.571741 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d2ae19cc80194363b8a9c96fbafb4ebe consensus state has no leader
I20251023 20:15:25.572184 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.572408 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.572615 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.572764 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.572963 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 473324b22da14618a84aa1136d081993 consensus state has no leader
I20251023 20:15:25.572986 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.573151 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.574086 23369 tablet_service.cc:1505] Processing CreateTablet for tablet bf858e0fc0634ef3868656d5dea6c82a (DEFAULT_TABLE table=default.Table_cd166f9dbcdb47ad988e2d387a6a1c5e [id=4c679b7dd63d4db2bb3ea41860cf80b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.574331 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 03f0984da42c49b683870b7c80b224a4 (DEFAULT_TABLE table=default.Table_f06b4a8de0ed4fee83a2d7e2a2eb3823 [id=e95b10e242404ffea012d67b902b54c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.574378 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.574538 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf858e0fc0634ef3868656d5dea6c82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.574086 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 44934f52cbeb448ba081b57cde783b2a (DEFAULT_TABLE table=default.Table_2dc2a5b423424735a1c370afde321275 [id=8f71d213abd24c078ff9db0f0b20c301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.575008 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.575196 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.575335 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.575508 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.575680 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44934f52cbeb448ba081b57cde783b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.575839 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.576135 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.576429 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.576718 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.576979 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.577229 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f0984da42c49b683870b7c80b224a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.577544 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.577817 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.578225 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.578372 23366 tablet_service.cc:1505] Processing CreateTablet for tablet f98f490c2caa49b8ae08241e5bb4df9b (DEFAULT_TABLE table=default.Table_a9bddf9d668f414dae97b08900de3f65 [id=78aea9dba8d94e5ca65f8c397f78395c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.578632 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.578727 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98f490c2caa49b8ae08241e5bb4df9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.579015 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.579424 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.579820 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.580915 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
W20251023 20:15:25.581172 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:25.581444 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b3a216aeebe649f79b6d873e1c59df37 consensus state has no leader
I20251023 20:15:25.581779 23634 tablet_service.cc:1505] Processing CreateTablet for tablet 03f0984da42c49b683870b7c80b224a4 (DEFAULT_TABLE table=default.Table_f06b4a8de0ed4fee83a2d7e2a2eb3823 [id=e95b10e242404ffea012d67b902b54c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.582134 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f0984da42c49b683870b7c80b224a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:25.584834 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a4d71f0cfdb6449fbd74b854c15326d6 consensus state has no leader
W20251023 20:15:25.585322 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 63bdf5c80f2542f2a982d1de157d848d consensus state has no leader
I20251023 20:15:25.585500 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 44934f52cbeb448ba081b57cde783b2a (DEFAULT_TABLE table=default.Table_2dc2a5b423424735a1c370afde321275 [id=8f71d213abd24c078ff9db0f0b20c301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.585920 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44934f52cbeb448ba081b57cde783b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.587806 24160 raft_consensus.cc:493] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.587966 24160 raft_consensus.cc:515] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.588630 24160 leader_election.cc:290] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:25.589488 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.589944 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.590276 23604 consensus_peers.cc:597] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:25.591109 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a708cd011802458aa0e2991fd4b4a2d3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.591646 23605 leader_election.cc:343] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.591883 24160 consensus_queue.cc:1048] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.593096 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.596810 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.597848 23650 tablet_service.cc:1505] Processing CreateTablet for tablet bf858e0fc0634ef3868656d5dea6c82a (DEFAULT_TABLE table=default.Table_cd166f9dbcdb47ad988e2d387a6a1c5e [id=4c679b7dd63d4db2bb3ea41860cf80b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.598249 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf858e0fc0634ef3868656d5dea6c82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.608348 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.608721 24339 consensus_queue.cc:1048] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.608927 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.609648 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.610136 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.610746 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.611241 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.611886 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.612366 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.613127 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet c6f3abcaa81c4646bc1432bcbd1d4fcf consensus state has no leader
I20251023 20:15:25.613171 24160 raft_consensus.cc:493] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.613317 24160 raft_consensus.cc:515] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.613824 24230 consensus_queue.cc:1048] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.613952 24160 leader_election.cc:290] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.614843 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:25.615337 23605 leader_election.cc:343] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.615684 23646 tablet_service.cc:1505] Processing CreateTablet for tablet f98f490c2caa49b8ae08241e5bb4df9b (DEFAULT_TABLE table=default.Table_a9bddf9d668f414dae97b08900de3f65 [id=78aea9dba8d94e5ca65f8c397f78395c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:25.615820 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 88f1df552f93427e91ca19c8170d69bd consensus state has no leader
I20251023 20:15:25.616619 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98f490c2caa49b8ae08241e5bb4df9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:25.620903 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4f47ddde9f3d4042bc84853332e21950 consensus state has no leader
I20251023 20:15:25.617599 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.621325 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 0150ab3778c34c77a7e772dd6e82b32f consensus state has no leader
I20251023 20:15:25.621834 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.622093 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.622291 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.622809 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 19c58a406d564b048fb16a85aa0b754b consensus state has no leader
I20251023 20:15:25.624853 24363 ts_tablet_manager.cc:1916] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.625564 24230 consensus_queue.cc:1048] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.631440 24363 ts_tablet_manager.cc:1929] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:25.631747 24363 log.cc:1199] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/aa794b3bec4b4b3993c814ea7cd02ee2
I20251023 20:15:25.632339 24363 ts_tablet_manager.cc:1950] T aa794b3bec4b4b3993c814ea7cd02ee2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.634001 24363 tablet_replica.cc:333] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:25.634326 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet aa794b3bec4b4b3993c814ea7cd02ee2 (table default.Table_083a66aed4484a38961dd79edd4d1c2e [id=a431259b5e254da6941241ffcf8ce597]) successfully deleted
W20251023 20:15:25.635020 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
W20251023 20:15:25.635236 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
W20251023 20:15:25.635396 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb915d114fc0421999d079c23cad8eb5 already in progress: creating tablet
W20251023 20:15:25.635535 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
W20251023 20:15:25.635701 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7e08dad1d0f643efb3a821a1b9196290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e08dad1d0f643efb3a821a1b9196290 already in progress: creating tablet
W20251023 20:15:25.635877 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
W20251023 20:15:25.640180 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2202632a57514c849760509360113078 consensus state has no leader
I20251023 20:15:25.640798 24341 consensus_queue.cc:1048] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.641175 24495 consensus_queue.cc:1048] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.635826 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.636196 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.636354 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.636762 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.637332 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.638728 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.639225 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.639770 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.641175 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.641645 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.644533 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.645500 24363 raft_consensus.cc:2243] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:25.645687 24363 raft_consensus.cc:2272] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.646170 24363 ts_tablet_manager.cc:1916] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.649437 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.650230 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.653115 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.653443 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.651916 24533 tablet.cc:1722] T 215e21ca449c481d8716783a65105101 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:25.654176 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.654693 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.654842 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.661336 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.661912 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.663033 24230 consensus_queue.cc:1048] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.693549 24547 tablet.cc:1722] T 215e21ca449c481d8716783a65105101 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:25.695178 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.695674 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.698712 23790 tablet_bootstrap.cc:492] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.705152 24230 raft_consensus.cc:493] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.705330 24230 raft_consensus.cc:515] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.705909 24230 leader_election.cc:290] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.706815 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.712098 24198 raft_consensus.cc:493] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.712283 24363 ts_tablet_manager.cc:1929] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:25.712587 24198 raft_consensus.cc:515] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.712718 24363 log.cc:1199] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:25.713310 24363 ts_tablet_manager.cc:1950] T 95574277dde44983bb0a43d2b5ad01d5 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.700718 23790 ts_tablet_manager.cc:1403] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.177s user 0.004s sys 0.011s
I20251023 20:15:25.714090 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e8eb7603c3e46f1b15cdff7a7bbedcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.714308 23396 raft_consensus.cc:2468] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:25.714941 23790 raft_consensus.cc:359] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.715157 23790 raft_consensus.cc:385] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.715158 23605 leader_election.cc:304] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:25.715230 23790 raft_consensus.cc:740] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.715452 23790 consensus_queue.cc:260] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.716107 23790 ts_tablet_manager.cc:1434] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:25.716449 23790 tablet_bootstrap.cc:492] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.716454 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 95574277dde44983bb0a43d2b5ad01d5 (table default.Table_2d1938985e9345eba1d4fb750b552e78 [id=8b537204d252421c8ef4084d1dd9226f]) successfully deleted
W20251023 20:15:25.717118 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a86a25659ec44d7c825783036342ed45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a86a25659ec44d7c825783036342ed45 already in progress: creating tablet
W20251023 20:15:25.717783 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f103bece2fa240c19c23ac51046c13ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f103bece2fa240c19c23ac51046c13ed already in progress: creating tablet
I20251023 20:15:25.717931 24198 leader_election.cc:290] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.718103 24198 raft_consensus.cc:2804] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.718227 24198 raft_consensus.cc:493] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.718281 23790 tablet_bootstrap.cc:654] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.718325 24198 raft_consensus.cc:3060] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:25.718541 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a8eaa2167f82475ca2f6a110aba288ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8eaa2167f82475ca2f6a110aba288ec already in progress: creating tablet
W20251023 20:15:25.719149 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 352812b56f8d4feb818a364d6adfd86d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352812b56f8d4feb818a364d6adfd86d already in progress: creating tablet
W20251023 20:15:25.719688 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 48436f3d8f1443db990a2d10b668b70a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48436f3d8f1443db990a2d10b668b70a already in progress: creating tablet
I20251023 20:15:25.720175 24363 tablet_replica.cc:333] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:25.720449 24363 raft_consensus.cc:2243] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:25.720825 24363 raft_consensus.cc:2272] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.722286 24363 ts_tablet_manager.cc:1916] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.727134 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532c8567db14901b383f42e5f04b4c2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.727396 24230 consensus_queue.cc:1048] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.727909 23667 raft_consensus.cc:2468] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:25.729594 24363 ts_tablet_manager.cc:1929] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:25.729802 24363 log.cc:1199] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:25.732203 24547 tablet.cc:1722] T 9c0aeac2f20c4ca48c130a0c851f290f P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:25.735309 23396 raft_consensus.cc:3060] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:25.738827 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 994a555b24e4432a84601c54d422aa5d consensus state has no leader
W20251023 20:15:25.739305 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 377de4ea0bd6489284fac6b0fab52c8c consensus state has no leader
W20251023 20:15:25.740449 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet cf38936bb12846bd81a2c79ffbd18ccb consensus state has no leader
I20251023 20:15:25.741556 24230 raft_consensus.cc:493] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.741796 24230 raft_consensus.cc:515] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.742674 24230 leader_election.cc:290] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.583316 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 03f0984da42c49b683870b7c80b224a4 (DEFAULT_TABLE table=default.Table_f06b4a8de0ed4fee83a2d7e2a2eb3823 [id=e95b10e242404ffea012d67b902b54c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.743574 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f0984da42c49b683870b7c80b224a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.743597 23790 tablet_bootstrap.cc:492] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.743884 23790 ts_tablet_manager.cc:1403] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20251023 20:15:25.730615 24363 ts_tablet_manager.cc:1950] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.746146 24363 tablet_replica.cc:333] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:25.746340 24363 raft_consensus.cc:2243] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:25.746517 24363 raft_consensus.cc:2272] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.746791 24174 consensus_queue.cc:1048] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.583827 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 44934f52cbeb448ba081b57cde783b2a (DEFAULT_TABLE table=default.Table_2dc2a5b423424735a1c370afde321275 [id=8f71d213abd24c078ff9db0f0b20c301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.748957 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44934f52cbeb448ba081b57cde783b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.584282 23516 tablet_service.cc:1505] Processing CreateTablet for tablet bf858e0fc0634ef3868656d5dea6c82a (DEFAULT_TABLE table=default.Table_cd166f9dbcdb47ad988e2d387a6a1c5e [id=4c679b7dd63d4db2bb3ea41860cf80b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.752681 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf858e0fc0634ef3868656d5dea6c82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.584771 23502 tablet_service.cc:1505] Processing CreateTablet for tablet f98f490c2caa49b8ae08241e5bb4df9b (DEFAULT_TABLE table=default.Table_a9bddf9d668f414dae97b08900de3f65 [id=78aea9dba8d94e5ca65f8c397f78395c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:25.753362 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98f490c2caa49b8ae08241e5bb4df9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:25.744655 23790 raft_consensus.cc:359] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.755020 23790 raft_consensus.cc:385] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.755124 23790 raft_consensus.cc:740] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.755388 23790 consensus_queue.cc:260] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.756047 23790 ts_tablet_manager.cc:1434] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251023 20:15:25.756165 24555 tablet.cc:1722] T 90d58bceecba442898a3d742fa49a4a2 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:25.756361 23790 tablet_bootstrap.cc:492] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.589656 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a708cd011802458aa0e2991fd4b4a2d3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.614547 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.629069 24472 tablet.cc:1722] T 215e21ca449c481d8716783a65105101 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:25.716125 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e8eb7603c3e46f1b15cdff7a7bbedcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:25.758037 23604 leader_election.cc:343] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.758150 23604 leader_election.cc:304] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:25.758543 23604 leader_election.cc:343] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.758632 23604 leader_election.cc:304] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.758846 24553 raft_consensus.cc:2749] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:25.758992 24553 raft_consensus.cc:2749] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:25.758972 23604 leader_election.cc:343] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.760186 23790 tablet_bootstrap.cc:654] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.762375 24472 tablet.cc:1722] T 9c0aeac2f20c4ca48c130a0c851f290f P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:25.762861 24363 ts_tablet_manager.cc:1916] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.768369 24472 tablet.cc:1722] T 90d58bceecba442898a3d742fa49a4a2 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:25.729552 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532c8567db14901b383f42e5f04b4c2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.774343 23790 tablet_bootstrap.cc:492] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.774540 23790 ts_tablet_manager.cc:1403] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20251023 20:15:25.774982 23786 ts_tablet_manager.cc:1403] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.273s user 0.002s sys 0.002s
I20251023 20:15:25.775436 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f (table default.Table_7b49c2bed47942a1b7cb25f8675a69ea [id=8cf42619d4b44a5990f7f63f1e1315ed]) successfully deleted
I20251023 20:15:25.775523 23790 raft_consensus.cc:359] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:25.775718 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6a54470d088748b0be22ab9db5a25f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a54470d088748b0be22ab9db5a25f87 already in progress: creating tablet
I20251023 20:15:25.775753 23790 raft_consensus.cc:385] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.775772 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.775825 23790 raft_consensus.cc:740] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.776144 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.776054 23790 consensus_queue.cc:260] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.776356 24230 raft_consensus.cc:493] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.776510 24230 raft_consensus.cc:515] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.776649 23790 ts_tablet_manager.cc:1434] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:25.776680 23786 raft_consensus.cc:359] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.776932 23786 raft_consensus.cc:385] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.776954 23790 tablet_bootstrap.cc:492] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.777159 24230 leader_election.cc:290] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.777551 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.777894 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.778172 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.778715 23790 tablet_bootstrap.cc:654] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.778533 24363 ts_tablet_manager.cc:1929] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:25.779309 24363 log.cc:1199] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:25.779631 24363 ts_tablet_manager.cc:1950] T d82c92e863b2470699cc822b8cb8c14d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.780664 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d82c92e863b2470699cc822b8cb8c14d (table default.Table_8ac87d9615a74c5c858be1d6c31772a6 [id=1740292e42a544e5bd5d834a281bddbc]) successfully deleted
W20251023 20:15:25.781287 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
W20251023 20:15:25.781847 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
I20251023 20:15:25.782091 24363 tablet_replica.cc:333] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:25.782286 24363 raft_consensus.cc:2243] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:25.782610 24363 raft_consensus.cc:2272] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.783447 24363 ts_tablet_manager.cc:1916] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.784425 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:25.785106 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
I20251023 20:15:25.789889 24363 ts_tablet_manager.cc:1929] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:25.790048 24363 log.cc:1199] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:25.790402 23790 tablet_bootstrap.cc:492] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.790413 24363 ts_tablet_manager.cc:1950] T cec51c54ea534b1da4b1d170e79d9f1c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.790580 23790 ts_tablet_manager.cc:1403] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251023 20:15:25.791363 23790 raft_consensus.cc:359] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.791575 23790 raft_consensus.cc:385] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.791671 23790 raft_consensus.cc:740] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.791908 23790 consensus_queue.cc:260] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.792371 24363 tablet_replica.cc:333] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:25.792464 24555 tablet.cc:1722] T dcbacd8e23bc4b129757d9faf89ce5b4 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:25.792623 24363 raft_consensus.cc:2243] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:25.792853 24363 raft_consensus.cc:2272] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.793390 24363 ts_tablet_manager.cc:1916] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.795753 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet cec51c54ea534b1da4b1d170e79d9f1c (table default.Table_21f0d4fca62a4309816edd74c6b137bf [id=be3ef152e4fd4281b5caf942c8d404ab]) successfully deleted
W20251023 20:15:25.795987 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 600f0110f8bd4b05983d5922d871c266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600f0110f8bd4b05983d5922d871c266 already in progress: creating tablet
W20251023 20:15:25.796198 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a636a4cdeb404e9f806d8e166ea57938 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a636a4cdeb404e9f806d8e166ea57938 already in progress: creating tablet
I20251023 20:15:25.799350 24363 ts_tablet_manager.cc:1929] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:25.799515 24363 log.cc:1199] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/523d72012c364457b9a2a299f2f789a1
I20251023 20:15:25.799882 24363 ts_tablet_manager.cc:1950] T 523d72012c364457b9a2a299f2f789a1 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.799742 23790 ts_tablet_manager.cc:1434] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:25.800604 23790 tablet_bootstrap.cc:492] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.801116 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 523d72012c364457b9a2a299f2f789a1 (table default.Table_7e026bfaccc4482dba743310ed0adb4c [id=7fdd0d3b02734ea89c0fce5e98fca82e]) successfully deleted
W20251023 20:15:25.801792 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9f422575c7c4251879130501ab721fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f422575c7c4251879130501ab721fd already in progress: creating tablet
W20251023 20:15:25.802399 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 83dae34129794f49b43038d3b1636ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83dae34129794f49b43038d3b1636ea0 already in progress: creating tablet
I20251023 20:15:25.803000 23790 tablet_bootstrap.cc:654] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:25.802968 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a9c5b69df4c04365b1ac1457fe8300cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9c5b69df4c04365b1ac1457fe8300cc already in progress: creating tablet
I20251023 20:15:25.803257 24363 tablet_replica.cc:333] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:25.803476 24363 raft_consensus.cc:2243] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:25.803861 24363 raft_consensus.cc:2272] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.804497 24363 ts_tablet_manager.cc:1916] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.810631 24363 ts_tablet_manager.cc:1929] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:25.810808 24363 log.cc:1199] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/f4778436aed64b53adb2bab54bcf263b
I20251023 20:15:25.811184 24363 ts_tablet_manager.cc:1950] T f4778436aed64b53adb2bab54bcf263b P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.811331 24259 raft_consensus.cc:493] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.811509 24259 raft_consensus.cc:515] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.812155 24259 leader_election.cc:290] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.812634 24363 tablet_replica.cc:333] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:25.812672 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 84327.1us
I20251023 20:15:25.812994 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e3d422a66f432789c0cf4a00e57af2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.813169 23667 raft_consensus.cc:2468] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:25.813324 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet f4778436aed64b53adb2bab54bcf263b (table default.Table_5abd17a4558f4f81a956cdf6df0d4524 [id=46e81c2c4c164452980264259fcff377]) successfully deleted
I20251023 20:15:25.813732 24220 consensus_queue.cc:1048] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:25.814390 23333 leader_election.cc:343] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.814539 23470 leader_election.cc:304] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:25.814821 24259 raft_consensus.cc:2804] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.814922 24259 raft_consensus.cc:493] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.814999 24259 raft_consensus.cc:3060] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.817612 24259 raft_consensus.cc:515] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.818557 24259 leader_election.cc:290] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.818992 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e3d422a66f432789c0cf4a00e57af2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:25.819164 23667 raft_consensus.cc:3060] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.821564 23667 raft_consensus.cc:2468] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:25.822045 23470 leader_election.cc:304] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:25.822372 23212 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc488f0beec14d0eb869b7474ec16205 has no consensus state
I20251023 20:15:25.823130 24363 raft_consensus.cc:2243] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:25.823294 24363 raft_consensus.cc:2272] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.824663 24363 ts_tablet_manager.cc:1916] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.826109 23215 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_4fb0e33f75594ca59d2ecb391873d804" } new_table_name: "default.Table_cb399f5d5c1542dca90056ed300477cd" modify_external_catalogs: true
I20251023 20:15:25.832978 24259 raft_consensus.cc:2804] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.833380 24259 raft_consensus.cc:697] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:25.834275 24259 consensus_queue.cc:237] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.837754 24363 ts_tablet_manager.cc:1929] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:25.837878 24363 log.cc:1199] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/7645c36c983f438f8b1223256fd3ad2c
I20251023 20:15:25.838223 24363 ts_tablet_manager.cc:1950] T 7645c36c983f438f8b1223256fd3ad2c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:25.839185 24363 tablet_replica.cc:333] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:25.839383 24363 raft_consensus.cc:2243] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:25.839407 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 7645c36c983f438f8b1223256fd3ad2c (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]) successfully deleted
I20251023 20:15:25.839529 24363 raft_consensus.cc:2272] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.840057 24363 ts_tablet_manager.cc:1916] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.843470 24198 raft_consensus.cc:515] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.844116 24198 leader_election.cc:290] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:25.844854 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e8eb7603c3e46f1b15cdff7a7bbedcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:25.845633 23604 leader_election.cc:343] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:25.777070 23786 raft_consensus.cc:740] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.858925 23786 consensus_queue.cc:260] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.862949 24259 raft_consensus.cc:493] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.863143 24259 raft_consensus.cc:515] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.864042 24259 leader_election.cc:290] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.864733 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.864948 23668 raft_consensus.cc:2468] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
W20251023 20:15:25.865412 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 50700.5us
I20251023 20:15:25.865470 23470 leader_election.cc:304] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:25.865927 24259 raft_consensus.cc:2804] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.866143 24220 raft_consensus.cc:493] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.866284 24220 raft_consensus.cc:515] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.866015 24259 raft_consensus.cc:493] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.866973 24259 raft_consensus.cc:3060] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.867085 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e8eb7603c3e46f1b15cdff7a7bbedcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:25.867272 23396 raft_consensus.cc:3060] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.869998 23396 raft_consensus.cc:2468] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:25.869935 24259 raft_consensus.cc:515] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.870631 23605 leader_election.cc:304] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.870740 24259 leader_election.cc:290] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.870934 24198 raft_consensus.cc:2804] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.871055 24198 raft_consensus.cc:697] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:25.871289 24198 consensus_queue.cc:237] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:25.871685 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 244fa327fad74aad9e57a7fbcbf908bb consensus state has no leader
W20251023 20:15:25.871917 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet c317dc31cb0448d985512e6ba98735be consensus state has no leader
I20251023 20:15:25.872614 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.873028 24220 leader_election.cc:290] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.874756 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.874872 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.875154 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.875360 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.876264 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.876909 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:25.877022 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
I20251023 20:15:25.877192 24349 tablet_replica.cc:333] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:25.877288 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
I20251023 20:15:25.877401 24349 raft_consensus.cc:2243] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:25.877437 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 95574277dde44983bb0a43d2b5ad01d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95574277dde44983bb0a43d2b5ad01d5 already in progress: creating tablet
I20251023 20:15:25.877585 24349 raft_consensus.cc:2272] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:25.877564 23218 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet f2a27706ecdb4ab78bb96c10a9f6757f consensus state has no leader
W20251023 20:15:25.877604 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7e08dad1d0f643efb3a821a1b9196290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e08dad1d0f643efb3a821a1b9196290 already in progress: creating tablet
I20251023 20:15:25.877988 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.878067 24220 raft_consensus.cc:493] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.878317 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.878249 24220 raft_consensus.cc:515] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.878485 24349 ts_tablet_manager.cc:1916] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.878859 24220 leader_election.cc:290] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.879717 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f47ddde9f3d4042bc84853332e21950" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.879931 23529 raft_consensus.cc:2393] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:25.880720 24533 tablet.cc:1722] T 9c0aeac2f20c4ca48c130a0c851f290f P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:25.880977 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8ba7ab28dfa4ac38e0b58ba2115495c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.881412 24339 raft_consensus.cc:493] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:25.881556 24339 raft_consensus.cc:515] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.882160 24339 leader_election.cc:290] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:25.883473 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f47ddde9f3d4042bc84853332e21950" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.883468 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8ba7ab28dfa4ac38e0b58ba2115495c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.888191 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:25.888361 23668 raft_consensus.cc:3060] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.889200 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.889617 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.890025 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.890380 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 6558265ff9e249c68123577958f1401a with delete_type TABLET_DATA_DELETED (replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.890836 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.891083 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.891433 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.891784 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.892148 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.892498 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.892874 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.893251 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.893395 23786 ts_tablet_manager.cc:1434] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.118s user 0.002s sys 0.000s
I20251023 20:15:25.893734 23786 tablet_bootstrap.cc:492] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:25.893934 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.894292 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.894692 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.895072 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.895450 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.895529 23786 tablet_bootstrap.cc:654] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.895913 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.896366 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.898213 23787 tablet_bootstrap.cc:492] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.898375 23787 ts_tablet_manager.cc:1403] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.364s user 0.026s sys 0.065s
I20251023 20:15:25.899097 23787 raft_consensus.cc:359] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.899272 23787 raft_consensus.cc:385] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.899355 23787 raft_consensus.cc:740] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.899592 23787 consensus_queue.cc:260] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:25.900238 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 975fc4848c8f4db4bfd2dad9ac620d83 consensus state has no leader
I20251023 20:15:25.900490 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.900751 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.900858 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.901242 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.901540 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.901724 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.901908 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.902143 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:25.902123 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3a7306245d7b41cfab795450761144f5 consensus state has no leader
I20251023 20:15:25.902755 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.902940 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.903082 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.903225 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.903342 24349 ts_tablet_manager.cc:1929] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:25.903355 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.903832 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.903861 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.904114 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:25.904184 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.904455 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.904470 23787 ts_tablet_manager.cc:1434] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20251023 20:15:25.904734 23219 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 188) took 1855 ms (client timeout 1000 ms). Trace:
W20251023 20:15:25.905022 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f already in progress: creating tablet
I20251023 20:15:25.905588 23787 tablet_bootstrap.cc:492] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.903457 24349 log.cc:1199] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/f1a6fe726ef844829a7ffec9d4494637
I20251023 20:15:25.907557 24349 ts_tablet_manager.cc:1950] T f1a6fe726ef844829a7ffec9d4494637 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:25.910697 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet f1a6fe726ef844829a7ffec9d4494637 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]) successfully deleted
W20251023 20:15:25.912029 23204 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet efad9b4a8e2e446f917df7ca26fbf789 consensus state has no leader
W20251023 20:15:25.912395 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
W20251023 20:15:25.912967 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:25.913321 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 48436f3d8f1443db990a2d10b668b70a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48436f3d8f1443db990a2d10b668b70a already in progress: creating tablet
W20251023 20:15:25.913637 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet aa794b3bec4b4b3993c814ea7cd02ee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa794b3bec4b4b3993c814ea7cd02ee2 already in progress: creating tablet
W20251023 20:15:25.913986 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a86a25659ec44d7c825783036342ed45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a86a25659ec44d7c825783036342ed45 already in progress: creating tablet
I20251023 20:15:25.914359 24349 tablet_replica.cc:333] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:25.914610 24349 raft_consensus.cc:2243] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:25.914851 24349 raft_consensus.cc:2272] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:25.915445 24349 ts_tablet_manager.cc:1916] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:25.924178 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e1b4748c0244b258960b1025da47562" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.924525 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:25.924896 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e3d422a66f432789c0cf4a00e57af2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.925165 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e3d422a66f432789c0cf4a00e57af2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:25.925271 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:25.925427 23399 raft_consensus.cc:2468] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:25.925680 23396 raft_consensus.cc:2468] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:25.925825 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:25.926034 23399 raft_consensus.cc:3060] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:25.926790 24472 tablet.cc:1722] T dcbacd8e23bc4b129757d9faf89ce5b4 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:25.928761 23399 raft_consensus.cc:2468] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:25.929119 23787 tablet_bootstrap.cc:654] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.930182 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e1b4748c0244b258960b1025da47562" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:25.930387 23669 raft_consensus.cc:2468] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:25.930663 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:25.904855 23219 rpcz_store.cc:269] 1023 20:15:24.049068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:24.049151 (+ 83us) service_pool.cc:225] Handling call
1023 20:15:25.904694 (+1855543us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:24.518790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:24.518925 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:24.518933 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:24.518936 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:24.519038 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:24.519051 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:24.519057 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:24.519120 (+ 63us) tablet.cc:699] Row locks acquired
1023 20:15:24.519122 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:24.519158 (+ 36us) write_op.cc:270] Start()
1023 20:15:24.519191 (+ 33us) write_op.cc:276] Timestamp: P: 1761250524519136 usec, L: 0
1023 20:15:24.519194 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:24.519378 (+ 184us) log.cc:844] Serialized 536 byte log entry
1023 20:15:24.523166 (+ 3788us) op_driver.cc:464] REPLICATION: finished
1023 20:15:24.523323 (+ 157us) write_op.cc:301] APPLY: starting
1023 20:15:24.523340 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:24.523398 (+ 58us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:24.523452 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:24.523538 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:24.523584 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:24.523592 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:24.523731 (+ 139us) log.cc:844] Serialized 23 byte log entry
1023 20:15:24.523806 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:24.523824 (+ 18us) write_op.cc:454] Released schema lock
1023 20:15:24.523831 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:24.523905 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:24.523909 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":137383,"reported_tablets":235,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":577,"apply.run_wall_time_us":577,"num_ops":1,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":700,"prepare.run_wall_time_us":700,"raft.queue_time_us":862,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":242,"replication_time_us":3942}]]}
I20251023 20:15:25.936244 23219 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_6d990f17621b44f78107cef2a0994e8e" } modify_external_catalogs: true
I20251023 20:15:25.936491 23219 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_6d990f17621b44f78107cef2a0994e8e" } modify_external_catalogs: true
I20251023 20:15:25.938963 23787 tablet_bootstrap.cc:492] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.938915 23333 leader_election.cc:304] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:25.939229 24220 raft_consensus.cc:2804] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:25.939330 24220 raft_consensus.cc:493] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:25.939426 24220 raft_consensus.cc:3060] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:25.939661 23333 leader_election.cc:343] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: Tablet not RUNNING: STOPPED
I20251023 20:15:25.951113 24220 raft_consensus.cc:515] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.952011 24220 leader_election.cc:290] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:25.953603 23218 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet aa59141039de402e9ccce427604ecb31 consensus state has no leader
I20251023 20:15:25.959465 23786 tablet_bootstrap.cc:492] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:25.963749 23786 ts_tablet_manager.cc:1403] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.000s
I20251023 20:15:25.964232 23790 tablet_bootstrap.cc:492] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.964407 23790 ts_tablet_manager.cc:1403] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.164s user 0.021s sys 0.012s
I20251023 20:15:25.965224 23790 raft_consensus.cc:359] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.965432 23790 raft_consensus.cc:385] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.965521 23790 raft_consensus.cc:740] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.965767 23790 consensus_queue.cc:260] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.973968 23790 ts_tablet_manager.cc:1434] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:25.974346 23790 tablet_bootstrap.cc:492] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.976189 23790 tablet_bootstrap.cc:654] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.976754 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.977330 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.939100 23787 ts_tablet_manager.cc:1403] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.001s
I20251023 20:15:25.977694 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.978106 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.978475 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.978466 23787 raft_consensus.cc:359] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.978646 23787 raft_consensus.cc:385] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.978814 23787 raft_consensus.cc:740] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.979128 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.979058 23787 consensus_queue.cc:260] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.979971 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.980093 24198 consensus_queue.cc:1048] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:25.980347 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.980669 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.981189 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.981523 23787 ts_tablet_manager.cc:1434] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:25.981554 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.981817 23787 tablet_bootstrap.cc:492] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.981935 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.982467 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.982846 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.983220 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.983711 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.984056 23787 tablet_bootstrap.cc:654] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.984069 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.984455 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.984793 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.985196 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.985539 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.985697 23790 tablet_bootstrap.cc:492] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:25.985854 23790 ts_tablet_manager.cc:1403] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20251023 20:15:25.985957 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.986291 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.948238 23668 raft_consensus.cc:2468] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:25.986730 23790 raft_consensus.cc:359] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.986929 23790 raft_consensus.cc:385] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.987020 23790 raft_consensus.cc:740] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.987097 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet c95db7b5b9fd46acae8ffa7119915907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.987211 23790 consensus_queue.cc:260] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.987507 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.988082 23790 ts_tablet_manager.cc:1434] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:25.988406 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.988727 23790 tablet_bootstrap.cc:492] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:25.989234 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.989733 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.990061 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.990422 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:25.990579 23790 tablet_bootstrap.cc:654] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:25.992442 23470 leader_election.cc:304] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:25.992697 24259 raft_consensus.cc:2804] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:25.993134 24259 raft_consensus.cc:697] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:25.993407 24259 consensus_queue.cc:237] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:25.994570 23787 tablet_bootstrap.cc:492] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:25.994755 23787 ts_tablet_manager.cc:1403] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20251023 20:15:25.995442 23787 raft_consensus.cc:359] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.995667 23787 raft_consensus.cc:385] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:25.995754 23787 raft_consensus.cc:740] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:25.996013 23787 consensus_queue.cc:260] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:25.996532 23787 ts_tablet_manager.cc:1434] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:25.996932 23787 tablet_bootstrap.cc:492] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.999408 23787 tablet_bootstrap.cc:654] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.000921 23790 tablet_bootstrap.cc:492] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.001165 23790 ts_tablet_manager.cc:1403] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20251023 20:15:26.002004 23790 raft_consensus.cc:359] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.002063 24533 tablet.cc:1722] T 9e76f2e26dcd4cdfbf72de899f5a1458 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:26.002285 23790 raft_consensus.cc:385] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.002403 23790 raft_consensus.cc:740] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.002704 23790 consensus_queue.cc:260] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.003526 23790 ts_tablet_manager.cc:1434] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:26.003881 23790 tablet_bootstrap.cc:492] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.005470 23790 tablet_bootstrap.cc:654] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.008069 23397 raft_consensus.cc:3060] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.008280 24472 tablet.cc:1722] T 9e76f2e26dcd4cdfbf72de899f5a1458 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:26.010218 23787 tablet_bootstrap.cc:492] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.010318 23787 ts_tablet_manager.cc:1403] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20251023 20:15:26.010926 23787 raft_consensus.cc:359] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.011062 23787 raft_consensus.cc:385] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.011099 23787 raft_consensus.cc:740] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.011330 23787 consensus_queue.cc:260] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.011775 23787 ts_tablet_manager.cc:1434] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251023 20:15:26.011772 24198 consensus_queue.cc:1048] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.012068 23787 tablet_bootstrap.cc:492] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:25.998193 23786 raft_consensus.cc:359] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.013293 23786 raft_consensus.cc:385] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.013391 23786 raft_consensus.cc:740] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.013635 23786 consensus_queue.cc:260] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.014161 23786 ts_tablet_manager.cc:1434] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20251023 20:15:26.014468 23786 tablet_bootstrap.cc:492] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.014459 23787 tablet_bootstrap.cc:654] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.016067 23786 tablet_bootstrap.cc:654] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.016368 23790 tablet_bootstrap.cc:492] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.016516 23790 ts_tablet_manager.cc:1403] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20251023 20:15:26.017237 23790 raft_consensus.cc:359] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.017410 23790 raft_consensus.cc:385] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.017514 23790 raft_consensus.cc:740] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.017753 23790 consensus_queue.cc:260] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.018497 23790 ts_tablet_manager.cc:1434] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:26.019068 23790 tablet_bootstrap.cc:492] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.020740 23790 tablet_bootstrap.cc:654] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.021318 24363 ts_tablet_manager.cc:1929] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:26.021471 24363 log.cc:1199] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a8ba7ab28dfa4ac38e0b58ba2115495c
I20251023 20:15:26.021849 24363 ts_tablet_manager.cc:1950] T a8ba7ab28dfa4ac38e0b58ba2115495c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.025622 23787 tablet_bootstrap.cc:492] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.025753 23787 ts_tablet_manager.cc:1403] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251023 20:15:26.025907 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.025688 24174 raft_consensus.cc:493] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.026353 24174 raft_consensus.cc:515] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.026438 23787 raft_consensus.cc:359] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.026609 23787 raft_consensus.cc:385] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.026664 23787 raft_consensus.cc:740] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.026813 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.026871 23787 consensus_queue.cc:260] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.027251 24174 leader_election.cc:290] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.027419 23787 ts_tablet_manager.cc:1434] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:26.027408 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.028106 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "925e53d9ea6f4c7baa38551eb917828d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.028586 23790 tablet_bootstrap.cc:492] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:26.028590 23605 leader_election.cc:343] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.028754 23790 ts_tablet_manager.cc:1403] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20251023 20:15:26.029276 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.029681 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.029816 23790 raft_consensus.cc:359] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.030010 23790 raft_consensus.cc:385] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.030090 23790 raft_consensus.cc:740] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.030439 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.030370 23790 consensus_queue.cc:260] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.031199 23790 ts_tablet_manager.cc:1434] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:26.031728 23790 tablet_bootstrap.cc:492] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.031961 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.032845 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.033394 23786 tablet_bootstrap.cc:492] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.033542 23786 ts_tablet_manager.cc:1403] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20251023 20:15:26.033815 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.033926 23790 tablet_bootstrap.cc:654] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.034188 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "925e53d9ea6f4c7baa38551eb917828d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.035842 23787 tablet_bootstrap.cc:492] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.036758 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.034344 23786 raft_consensus.cc:359] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.037176 23786 raft_consensus.cc:385] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.037302 23786 raft_consensus.cc:740] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.037587 23786 consensus_queue.cc:260] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.038260 23786 ts_tablet_manager.cc:1434] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251023 20:15:26.038555 23786 tablet_bootstrap.cc:492] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.038872 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.000650 24349 ts_tablet_manager.cc:1929] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:26.039541 24349 log.cc:1199] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/6558265ff9e249c68123577958f1401a
I20251023 20:15:26.039909 24349 ts_tablet_manager.cc:1950] T 6558265ff9e249c68123577958f1401a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.041489 24349 tablet_replica.cc:333] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:26.041733 24349 raft_consensus.cc:2243] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.041911 24349 raft_consensus.cc:2272] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.042008 23786 tablet_bootstrap.cc:654] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.042472 24349 ts_tablet_manager.cc:1916] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.046144 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 6558265ff9e249c68123577958f1401a (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]) successfully deleted
W20251023 20:15:26.046502 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a8eaa2167f82475ca2f6a110aba288ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8eaa2167f82475ca2f6a110aba288ec already in progress: creating tablet
W20251023 20:15:26.046833 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 352812b56f8d4feb818a364d6adfd86d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352812b56f8d4feb818a364d6adfd86d already in progress: creating tablet
W20251023 20:15:26.047159 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6a54470d088748b0be22ab9db5a25f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a54470d088748b0be22ab9db5a25f87 already in progress: creating tablet
W20251023 20:15:26.047564 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d82c92e863b2470699cc822b8cb8c14d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d82c92e863b2470699cc822b8cb8c14d already in progress: creating tablet
W20251023 20:15:26.048004 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
W20251023 20:15:26.048303 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cec51c54ea534b1da4b1d170e79d9f1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cec51c54ea534b1da4b1d170e79d9f1c already in progress: creating tablet
I20251023 20:15:26.052592 24559 raft_consensus.cc:493] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.052740 24559 raft_consensus.cc:515] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.053386 24559 leader_election.cc:290] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.053648 24559 raft_consensus.cc:493] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.053750 24559 raft_consensus.cc:515] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.054205 24559 leader_election.cc:290] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.055655 23786 tablet_bootstrap.cc:492] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
W20251023 20:15:26.055759 23604 leader_election.cc:343] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.055817 23786 ts_tablet_manager.cc:1403] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251023 20:15:26.055886 24160 raft_consensus.cc:493] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.055976 24160 raft_consensus.cc:515] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.056317 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c8403b53f04a8e97ffe058e59b425c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.056442 24160 leader_election.cc:290] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.056607 23786 raft_consensus.cc:359] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.056783 23786 raft_consensus.cc:385] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.056840 23786 raft_consensus.cc:740] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.057101 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e0301fb5974ec5b6e8291230a736c2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.057050 23786 consensus_queue.cc:260] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:26.057041 23605 leader_election.cc:343] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.057662 23786 ts_tablet_manager.cc:1434] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:26.057297 23399 raft_consensus.cc:2468] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:26.057963 23786 tablet_bootstrap.cc:492] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.058873 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.058923 23604 leader_election.cc:304] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.059242 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.059643 23786 tablet_bootstrap.cc:654] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.059921 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.060101 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b6c3e7169154e3497dd3570dca0d745" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.060250 24553 raft_consensus.cc:2749] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.060072 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e0301fb5974ec5b6e8291230a736c2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.060436 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.060913 23605 leader_election.cc:304] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.061044 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.061638 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:26.061672 23604 leader_election.cc:343] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.062534 24553 raft_consensus.cc:2804] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.062616 24553 raft_consensus.cc:493] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.062618 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.062671 24553 raft_consensus.cc:3060] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.062963 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.063378 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.063678 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:26.063975 23604 leader_election.cc:343] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.065560 23787 tablet_bootstrap.cc:654] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.067648 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b6c3e7169154e3497dd3570dca0d745" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.067852 23396 raft_consensus.cc:2468] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:26.068445 23605 leader_election.cc:304] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.068912 24160 raft_consensus.cc:2804] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.068989 24160 raft_consensus.cc:493] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.069043 24160 raft_consensus.cc:3060] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.071645 24160 raft_consensus.cc:515] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.072216 24160 leader_election.cc:290] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:26.073123 23470 consensus_peers.cc:597] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 523d72012c364457b9a2a299f2f789a1. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:26.073261 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b6c3e7169154e3497dd3570dca0d745" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.073681 24553 raft_consensus.cc:515] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.074151 24553 leader_election.cc:290] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:26.074361 23604 leader_election.cc:343] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.074851 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e0301fb5974ec5b6e8291230a736c2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:26.075323 23604 leader_election.cc:343] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.076164 24259 consensus_queue.cc:1048] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.078068 24363 tablet_replica.cc:333] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.078406 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e0301fb5974ec5b6e8291230a736c2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.078462 24363 raft_consensus.cc:2243] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.078617 24363 raft_consensus.cc:2272] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.079348 23399 raft_consensus.cc:3060] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.079348 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b6c3e7169154e3497dd3570dca0d745" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.079612 24555 tablet.cc:1722] T 9e76f2e26dcd4cdfbf72de899f5a1458 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251023 20:15:26.079730 23396 raft_consensus.cc:3060] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.080340 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a8ba7ab28dfa4ac38e0b58ba2115495c (table default.Table_4e18a968658f470d870bc932102ebf00 [id=ad634b38921149658d6212b02c421514]) successfully deleted
I20251023 20:15:26.081043 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c8403b53f04a8e97ffe058e59b425c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:26.081481 23604 leader_election.cc:343] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.081576 23604 leader_election.cc:304] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.081859 24363 ts_tablet_manager.cc:1916] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.083231 23396 raft_consensus.cc:2468] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:26.081924 24174 raft_consensus.cc:2749] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.083774 23605 leader_election.cc:304] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.083988 24174 raft_consensus.cc:2804] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.084096 24174 raft_consensus.cc:697] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:26.084336 24174 consensus_queue.cc:237] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.086198 23786 tablet_bootstrap.cc:492] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.086333 23786 ts_tablet_manager.cc:1403] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20251023 20:15:26.087219 23786 raft_consensus.cc:359] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.087404 23786 raft_consensus.cc:385] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.087502 23786 raft_consensus.cc:740] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.087776 23786 consensus_queue.cc:260] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.088438 23786 ts_tablet_manager.cc:1434] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:26.091354 23399 raft_consensus.cc:2468] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:26.093398 23786 tablet_bootstrap.cc:492] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.095817 23786 tablet_bootstrap.cc:654] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.097272 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.099862 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.100842 23605 leader_election.cc:304] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.101436 24553 raft_consensus.cc:2804] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.101580 24553 raft_consensus.cc:697] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:26.101769 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.102977 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.104461 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.108026 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.109962 23786 tablet_bootstrap.cc:492] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.110143 23786 ts_tablet_manager.cc:1403] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251023 20:15:26.110356 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.110980 23786 raft_consensus.cc:359] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.111191 23786 raft_consensus.cc:385] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.111263 23786 raft_consensus.cc:740] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.111598 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.112377 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.114709 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.115653 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.111544 23786 consensus_queue.cc:260] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.116815 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.117165 23786 ts_tablet_manager.cc:1434] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:26.117491 23786 tablet_bootstrap.cc:492] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.118050 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.119065 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.119278 23786 tablet_bootstrap.cc:654] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.119995 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.120963 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.101848 24553 consensus_queue.cc:237] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.122643 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.124532 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.125805 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.126574 24160 consensus_queue.cc:1048] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.126703 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.127677 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:26.127784 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 170167us
I20251023 20:15:26.129209 23333 leader_election.cc:304] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:26.129536 23333 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.129820 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.130143 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.130674 24573 raft_consensus.cc:2804] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.131203 24573 raft_consensus.cc:493] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.131385 24573 raft_consensus.cc:3060] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:26.131738 23218 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 162) took 2104 ms (client timeout 1000 ms). Trace:
I20251023 20:15:26.131987 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.132824 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.134258 24573 raft_consensus.cc:515] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.134902 24573 leader_election.cc:290] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.135618 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.137404 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.138024 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.139683 23790 tablet_bootstrap.cc:492] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.139854 23790 ts_tablet_manager.cc:1403] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.108s user 0.006s sys 0.019s
I20251023 20:15:26.140719 23790 raft_consensus.cc:359] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.140918 23790 raft_consensus.cc:385] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.141010 23790 raft_consensus.cc:740] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.141289 23790 consensus_queue.cc:260] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.141942 23790 ts_tablet_manager.cc:1434] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:26.146027 23787 tablet_bootstrap.cc:492] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.146181 23787 ts_tablet_manager.cc:1403] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.005s
I20251023 20:15:26.144536 23790 tablet_bootstrap.cc:492] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.149490 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae388da9e4f94ba5af82a67088ab7097" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.149711 23670 raft_consensus.cc:2393] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
W20251023 20:15:26.144546 23218 rpcz_store.cc:269] 1023 20:15:24.027224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:24.027288 (+ 64us) service_pool.cc:225] Handling call
1023 20:15:24.632149 (+604861us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:24.632153 (+ 4us) write_op.cc:454] Released schema lock
1023 20:15:26.131697 (+1499544us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:24.626523 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:24.626636 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:24.626642 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:24.626644 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:24.626731 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:24.626743 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:24.626747 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:24.626800 (+ 53us) tablet.cc:699] Row locks acquired
1023 20:15:24.626803 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:24.626834 (+ 31us) write_op.cc:270] Start()
1023 20:15:24.626860 (+ 26us) write_op.cc:276] Timestamp: P: 1761250524626806 usec, L: 0
1023 20:15:24.626864 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:24.627006 (+ 142us) log.cc:844] Serialized 536 byte log entry
1023 20:15:24.629933 (+ 2927us) op_driver.cc:464] REPLICATION: finished
1023 20:15:24.630062 (+ 129us) write_op.cc:301] APPLY: starting
1023 20:15:24.630075 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:24.630110 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:24.630140 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:24.630188 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:24.630216 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:24.630223 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:24.630287 (+ 64us) log.cc:844] Serialized 23 byte log entry
1023 20:15:24.630334 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:24.630350 (+ 16us) write_op.cc:454] Released schema lock
1023 20:15:24.630356 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":24577,"reported_tablets":254,"child_traces":[["op",{"apply.queue_time_us":87,"apply.run_cpu_time_us":338,"apply.run_wall_time_us":338,"num_ops":1,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":572,"raft.queue_time_us":127,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":191,"replication_time_us":3044}]]}
I20251023 20:15:26.153146 23790 tablet_bootstrap.cc:654] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.153771 24363 ts_tablet_manager.cc:1929] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.153894 24363 log.cc:1199] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/9d1ddde7e3134e23b3a9356523a93753
I20251023 20:15:26.154531 24349 ts_tablet_manager.cc:1929] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.155424 23787 raft_consensus.cc:359] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.155822 23787 raft_consensus.cc:385] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.155892 23787 raft_consensus.cc:740] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.153811 23218 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_05604aaaf6604c4c8f929ac257e5824e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:26.156142 23787 consensus_queue.cc:260] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.157152 23218 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05604aaaf6604c4c8f929ac257e5824e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:26.157480 24349 log.cc:1199] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/ef1229b9e0384a82811f244d2bf4c8ce
I20251023 20:15:26.158257 24349 ts_tablet_manager.cc:1950] T ef1229b9e0384a82811f244d2bf4c8ce P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.161269 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet ef1229b9e0384a82811f244d2bf4c8ce (table default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]) successfully deleted
W20251023 20:15:26.161746 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 600f0110f8bd4b05983d5922d871c266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600f0110f8bd4b05983d5922d871c266 already in progress: creating tablet
I20251023 20:15:26.162957 23787 ts_tablet_manager.cc:1434] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251023 20:15:26.163296 23787 tablet_bootstrap.cc:492] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.166709 23787 tablet_bootstrap.cc:654] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.166756 24533 tablet.cc:1722] T dcbacd8e23bc4b129757d9faf89ce5b4 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:26.161031 24349 tablet_replica.cc:333] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:26.164059 24363 ts_tablet_manager.cc:1950] T 9d1ddde7e3134e23b3a9356523a93753 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.165977 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532c8567db14901b383f42e5f04b4c2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.168854 23532 raft_consensus.cc:3060] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.174364 23532 raft_consensus.cc:2468] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.175014 23218 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 44) took 1012 ms. Trace:
I20251023 20:15:26.177688 24349 raft_consensus.cc:2243] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:26.178424 23218 rpcz_store.cc:276] 1023 20:15:25.162816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.153550 (+990734us) service_pool.cc:225] Handling call
1023 20:15:26.158714 (+ 5164us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:26.159227 (+ 513us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:26.174731 (+ 15504us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:26.174870 (+ 139us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:26.174974 (+ 104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.166721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.166854 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.166865 (+ 11us) write_op.cc:435] Acquired schema lock
1023 20:15:26.166867 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:26.166966 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.166981 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.166986 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.167061 (+ 75us) tablet.cc:699] Row locks acquired
1023 20:15:26.167063 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:26.167103 (+ 40us) write_op.cc:270] Start()
1023 20:15:26.167136 (+ 33us) write_op.cc:276] Timestamp: P: 1761250526167074 usec, L: 0
1023 20:15:26.167140 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.167329 (+ 189us) log.cc:844] Serialized 433 byte log entry
1023 20:15:26.170556 (+ 3227us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.171332 (+ 776us) write_op.cc:301] APPLY: starting
1023 20:15:26.171351 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.171404 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.171442 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.171566 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.171615 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:26.171623 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:26.171713 (+ 90us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.171960 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.171983 (+ 23us) write_op.cc:454] Released schema lock
1023 20:15:26.171991 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.172064 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.172068 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":716,"apply.run_cpu_time_us":554,"apply.run_wall_time_us":727,"num_ops":2,"prepare.queue_time_us":6962,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":726,"raft.queue_time_us":953,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":223,"replication_time_us":3386}]]}
I20251023 20:15:26.181752 24573 raft_consensus.cc:493] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.181969 24573 raft_consensus.cc:515] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.182626 24573 leader_election.cc:290] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.183007 23790 tablet_bootstrap.cc:492] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.183187 23790 ts_tablet_manager.cc:1403] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.005s
I20251023 20:15:26.184114 23790 raft_consensus.cc:359] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.184334 23790 raft_consensus.cc:385] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.184404 23790 raft_consensus.cc:740] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.183209 24349 raft_consensus.cc:2272] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.184669 23790 consensus_queue.cc:260] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.186005 23790 ts_tablet_manager.cc:1434] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251023 20:15:26.186431 24349 ts_tablet_manager.cc:1916] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.186901 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2532c8567db14901b383f42e5f04b4c2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:26.187088 23666 raft_consensus.cc:3060] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.187217 23218 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_e6e2be420a6342bc9d1311bd4fe6080c" } modify_external_catalogs: true
I20251023 20:15:26.187438 23218 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_e6e2be420a6342bc9d1311bd4fe6080c" } modify_external_catalogs: true
I20251023 20:15:26.187630 23790 tablet_bootstrap.cc:492] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.192042 24573 raft_consensus.cc:493] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.192202 24573 raft_consensus.cc:515] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.192795 24573 leader_election.cc:290] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.195240 24533 tablet.cc:1722] T 90d58bceecba442898a3d742fa49a4a2 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:26.200542 23787 tablet_bootstrap.cc:492] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.200685 23787 ts_tablet_manager.cc:1403] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.012s
I20251023 20:15:26.195533 23666 raft_consensus.cc:2468] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.202672 23790 tablet_bootstrap.cc:654] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.210110 24349 ts_tablet_manager.cc:1929] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.210271 24349 log.cc:1199] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/8da82e6d24b14a129b91bb5298139d07
I20251023 20:15:26.210325 23333 leader_election.cc:304] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.210727 24349 ts_tablet_manager.cc:1950] T 8da82e6d24b14a129b91bb5298139d07 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.212000 24363 tablet_replica.cc:333] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.212195 24363 raft_consensus.cc:2243] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.212312 24363 raft_consensus.cc:2272] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.212608 24573 raft_consensus.cc:2749] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.212793 24363 ts_tablet_manager.cc:1916] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.215234 24600 raft_consensus.cc:493] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.215358 24600 raft_consensus.cc:515] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.216531 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 9d1ddde7e3134e23b3a9356523a93753 (table default.Table_a1332462902c4ea5971afeb4c3688865 [id=f27defb723394b4db220b278b92e6ae8]) successfully deleted
I20251023 20:15:26.216564 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "951b2276d3324c2d999c8d7fb97767bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:26.217110 23604 leader_election.cc:343] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.217630 23787 raft_consensus.cc:359] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.217787 23787 raft_consensus.cc:385] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.217832 23787 raft_consensus.cc:740] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.217994 23787 consensus_queue.cc:260] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.218617 24573 raft_consensus.cc:493] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.218770 24573 raft_consensus.cc:515] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.218838 24600 leader_election.cc:290] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.219412 24573 leader_election.cc:290] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.221200 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 8da82e6d24b14a129b91bb5298139d07 (table default.Table_58731fce9c224ff0b59d4b241110e518 [id=9ec90b724d89464a90821dd52282f77c]) successfully deleted
I20251023 20:15:26.221715 23787 ts_tablet_manager.cc:1434] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
W20251023 20:15:26.221985 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a636a4cdeb404e9f806d8e166ea57938 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a636a4cdeb404e9f806d8e166ea57938 already in progress: creating tablet
I20251023 20:15:26.222759 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "951b2276d3324c2d999c8d7fb97767bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.223013 23397 raft_consensus.cc:2468] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:26.223498 23605 leader_election.cc:304] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.223726 23787 tablet_bootstrap.cc:492] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.223922 24160 raft_consensus.cc:2804] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.224000 24160 raft_consensus.cc:493] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.224057 24160 raft_consensus.cc:3060] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.225550 23787 tablet_bootstrap.cc:654] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.226871 24160 raft_consensus.cc:515] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.227903 24160 leader_election.cc:290] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.228175 24349 tablet_replica.cc:333] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:26.228355 24363 ts_tablet_manager.cc:1929] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:26.228411 24349 raft_consensus.cc:2243] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:26.228581 24363 log.cc:1199] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/3a4c8dcbdfcf4a15b8a4194ea04b93a0
I20251023 20:15:26.228605 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "951b2276d3324c2d999c8d7fb97767bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.228775 24349 pending_rounds.cc:70] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Trying to abort 1 pending ops.
I20251023 20:15:26.229009 24363 ts_tablet_manager.cc:1950] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.228974 24349 pending_rounds.cc:77] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082151034671104 op_type: NO_OP noop_request { }
I20251023 20:15:26.229147 24349 raft_consensus.cc:2889] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20251023 20:15:26.229241 23604 leader_election.cc:343] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.229295 24349 raft_consensus.cc:2272] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.230165 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 (table default.Table_e926815d67914fed915b184838f05b72 [id=265a9707ddea4ee1a19aac6033883822]) successfully deleted
I20251023 20:15:26.230254 24349 ts_tablet_manager.cc:1916] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.230456 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "951b2276d3324c2d999c8d7fb97767bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.230657 24363 tablet_replica.cc:333] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.230679 23397 raft_consensus.cc:3060] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.231117 24363 raft_consensus.cc:2243] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.231765 24363 raft_consensus.cc:2272] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.232815 24363 ts_tablet_manager.cc:1916] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.233981 23786 tablet_bootstrap.cc:492] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.234134 23786 ts_tablet_manager.cc:1403] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.117s user 0.001s sys 0.005s
I20251023 20:15:26.234601 23397 raft_consensus.cc:2468] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:26.234997 23786 raft_consensus.cc:359] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.235046 23605 leader_election.cc:304] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.235157 23786 raft_consensus.cc:385] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.235230 23786 raft_consensus.cc:740] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.235458 23786 consensus_queue.cc:260] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.235783 24160 raft_consensus.cc:2804] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.236277 23786 ts_tablet_manager.cc:1434] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:26.236299 24160 raft_consensus.cc:697] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:26.236591 23786 tablet_bootstrap.cc:492] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.236716 24160 consensus_queue.cc:237] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.238632 23786 tablet_bootstrap.cc:654] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:26.213106 23333 leader_election.cc:343] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.247354 23333 leader_election.cc:304] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.247681 24573 raft_consensus.cc:2749] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.248992 24555 tablet.cc:1722] T 4e72922aae98430da968874f44603317 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:26.256037 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:26.256306 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:26.256505 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
W20251023 20:15:26.261056 23333 leader_election.cc:343] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.261615 23333 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.262903 23333 leader_election.cc:304] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.263200 23787 tablet_bootstrap.cc:492] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.263348 23787 ts_tablet_manager.cc:1403] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.001s
I20251023 20:15:26.264194 23787 raft_consensus.cc:359] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.264371 23787 raft_consensus.cc:385] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.264416 23787 raft_consensus.cc:740] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.264492 24186 raft_consensus.cc:2749] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:26.267089 23333 consensus_peers.cc:597] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:26.267031 24555 tablet.cc:1722] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:26.267951 23333 leader_election.cc:304] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:26.268488 24573 raft_consensus.cc:2804] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.268651 24573 raft_consensus.cc:697] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.268939 24573 consensus_queue.cc:237] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.279670 24349 ts_tablet_manager.cc:1929] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.279837 24349 log.cc:1199] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/523d72012c364457b9a2a299f2f789a1
I20251023 20:15:26.280251 24349 ts_tablet_manager.cc:1950] T 523d72012c364457b9a2a299f2f789a1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.280243 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:26.280505 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:26.281430 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 523d72012c364457b9a2a299f2f789a1 (table default.Table_7e026bfaccc4482dba743310ed0adb4c [id=7fdd0d3b02734ea89c0fce5e98fca82e]) successfully deleted
I20251023 20:15:26.282243 23787 consensus_queue.cc:260] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.282488 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f103bece2fa240c19c23ac51046c13ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f103bece2fa240c19c23ac51046c13ed already in progress: creating tablet
I20251023 20:15:26.282769 23787 ts_tablet_manager.cc:1434] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251023 20:15:26.283069 23787 tablet_bootstrap.cc:492] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.283345 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e1b4748c0244b258960b1025da47562" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:26.283499 23669 raft_consensus.cc:3060] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.284880 23787 tablet_bootstrap.cc:654] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.286231 23669 raft_consensus.cc:2468] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.287115 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:26.287995 24363 ts_tablet_manager.cc:1929] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:26.289124 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:26.290493 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
W20251023 20:15:26.291951 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9f422575c7c4251879130501ab721fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f422575c7c4251879130501ab721fd already in progress: creating tablet
I20251023 20:15:26.292253 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:26.293224 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:26.293584 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:26.289688 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.293727 24363 log.cc:1199] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/33b5efb30b5c491c8eafe3ac25688843
I20251023 20:15:26.293817 23669 raft_consensus.cc:2468] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.291020 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5346a221c78342359eb6e418bd12cd24" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.294035 23668 raft_consensus.cc:2468] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.295248 23790 tablet_bootstrap.cc:492] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.295399 23790 ts_tablet_manager.cc:1403] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.008s
I20251023 20:15:26.296406 23790 raft_consensus.cc:359] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.296586 23790 raft_consensus.cc:385] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.296684 23790 raft_consensus.cc:740] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.296882 23790 consensus_queue.cc:260] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.297700 23333 leader_election.cc:304] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.298339 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:26.298287 23790 ts_tablet_manager.cc:1434] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:26.298676 24573 raft_consensus.cc:2749] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.298646 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.299672 24259 consensus_queue.cc:1048] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.299758 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:26.300141 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:26.300344 23790 tablet_bootstrap.cc:492] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.300377 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:26.300922 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:26.301059 23786 tablet_bootstrap.cc:492] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.301244 23786 ts_tablet_manager.cc:1403] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.001s
I20251023 20:15:26.302157 23790 tablet_bootstrap.cc:654] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.302423 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:26.305495 23333 leader_election.cc:304] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.305761 24573 raft_consensus.cc:2804] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.305893 24573 raft_consensus.cc:697] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.306146 24573 consensus_queue.cc:237] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.306533 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:26.307119 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:26.307983 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:26.308590 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:26.308830 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:26.309027 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:26.309402 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:26.309679 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:26.309639 23786 raft_consensus.cc:359] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.309847 23786 raft_consensus.cc:385] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.309930 23786 raft_consensus.cc:740] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.310215 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95574277dde44983bb0a43d2b5ad01d5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.310429 23528 raft_consensus.cc:2468] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.310490 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e1b4748c0244b258960b1025da47562" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.310667 23529 raft_consensus.cc:3060] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.311154 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:26.311373 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:26.312402 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:26.312633 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:26.312865 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
W20251023 20:15:26.313017 23333 leader_election.cc:343] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:26.313211 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:26.313338 23333 leader_election.cc:304] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.313642 24472 tablet.cc:1722] T 4e72922aae98430da968874f44603317 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:26.313814 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:26.314054 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5346a221c78342359eb6e418bd12cd24" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.314302 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:26.316116 23529 raft_consensus.cc:2468] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.316159 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:26.317004 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:26.317225 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:26.317495 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]). Expected version 1 got 0
I20251023 20:15:26.318089 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]). Expected version 1 got 0
I20251023 20:15:26.318442 23217 catalog_manager.cc:5649] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:26.318975 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]). Expected version 3 got 0
I20251023 20:15:26.318933 24472 tablet.cc:1722] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:26.319270 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971]). Expected version 1 got 0
I20251023 20:15:26.319743 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:26.320159 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:26.320488 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:26.320729 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]). Expected version 2 got 0
I20251023 20:15:26.320974 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]). Expected version 2 got 0
W20251023 20:15:26.321251 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c97e2e0f839a4ef389fe100b6a81301e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c97e2e0f839a4ef389fe100b6a81301e already in progress: creating tablet
I20251023 20:15:26.321462 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]). Expected version 4 got 0
I20251023 20:15:26.321839 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]). Expected version 3 got 0
W20251023 20:15:26.322065 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cbacaf2fea24405caaf84d59b625fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbacaf2fea24405caaf84d59b625fca3 already in progress: creating tablet
I20251023 20:15:26.322243 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
W20251023 20:15:26.322914 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb915d114fc0421999d079c23cad8eb5 already in progress: creating tablet
I20251023 20:15:26.322993 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:26.323204 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]). Expected version 2 got 0
I20251023 20:15:26.323778 23786 consensus_queue.cc:260] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:26.325050 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
I20251023 20:15:26.325601 23333 leader_election.cc:304] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:26.325814 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 83dae34129794f49b43038d3b1636ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83dae34129794f49b43038d3b1636ea0 already in progress: creating tablet
I20251023 20:15:26.326344 23786 ts_tablet_manager.cc:1434] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
W20251023 20:15:26.326951 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
I20251023 20:15:26.327054 24160 raft_consensus.cc:493] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.327172 24160 raft_consensus.cc:515] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.327749 24160 leader_election.cc:290] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:26.327906 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a9c5b69df4c04365b1ac1457fe8300cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9c5b69df4c04365b1ac1457fe8300cc already in progress: creating tablet
I20251023 20:15:26.328099 24297 raft_consensus.cc:2804] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.328224 24297 raft_consensus.cc:493] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.328320 24297 raft_consensus.cc:3060] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.328303 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a7306245d7b41cfab795450761144f5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:26.328732 23604 leader_election.cc:343] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.329463 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a7306245d7b41cfab795450761144f5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:26.329850 23333 consensus_peers.cc:597] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef1229b9e0384a82811f244d2bf4c8ce. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:26.329962 23532 raft_consensus.cc:3060] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:26.330102 23333 leader_election.cc:343] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.330346 24495 raft_consensus.cc:2804] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.330454 24495 raft_consensus.cc:493] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.330546 24495 raft_consensus.cc:3060] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.331619 23333 leader_election.cc:304] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:26.332139 23333 leader_election.cc:343] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.332547 23790 tablet_bootstrap.cc:492] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.332787 23790 ts_tablet_manager.cc:1403] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20251023 20:15:26.333190 24339 raft_consensus.cc:2804] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.333294 24339 raft_consensus.cc:493] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.333386 24339 raft_consensus.cc:3060] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.333760 23790 raft_consensus.cc:359] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.333954 23790 raft_consensus.cc:385] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.334029 23790 raft_consensus.cc:740] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.334254 23790 consensus_queue.cc:260] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.334470 23605 leader_election.cc:343] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.334575 23605 leader_election.cc:304] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.334801 23790 ts_tablet_manager.cc:1434] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:26.334945 24160 raft_consensus.cc:2749] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.335129 23790 tablet_bootstrap.cc:492] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.336114 24339 raft_consensus.cc:515] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.336709 24339 leader_election.cc:290] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.337303 24607 tablet.cc:1722] T 5eab3e95be0f41dcb5a8ec63e91f5d8e P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:26.337860 24533 tablet.cc:1722] T 4e72922aae98430da968874f44603317 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251023 20:15:26.344930 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1a64a7c2e1f242e1b54243489622bae5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a64a7c2e1f242e1b54243489622bae5 already in progress: creating tablet
W20251023 20:15:26.345705 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet bb475e303d07491bbc19dc3f628bceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb475e303d07491bbc19dc3f628bceb7 already in progress: creating tablet
W20251023 20:15:26.346231 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 925e53d9ea6f4c7baa38551eb917828d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 925e53d9ea6f4c7baa38551eb917828d already in progress: creating tablet
W20251023 20:15:26.346769 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a708cd011802458aa0e2991fd4b4a2d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a708cd011802458aa0e2991fd4b4a2d3 already in progress: creating tablet
I20251023 20:15:26.347927 23786 tablet_bootstrap.cc:492] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.337055 23790 tablet_bootstrap.cc:654] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:26.351622 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 4b6c3e7169154e3497dd3570dca0d745 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b6c3e7169154e3497dd3570dca0d745 already in progress: creating tablet
I20251023 20:15:26.351898 24297 raft_consensus.cc:515] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.353438 23787 tablet_bootstrap.cc:492] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:26.353418 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 951b2276d3324c2d999c8d7fb97767bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 951b2276d3324c2d999c8d7fb97767bc already in progress: creating tablet
I20251023 20:15:26.353639 23787 ts_tablet_manager.cc:1403] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.071s user 0.000s sys 0.003s
W20251023 20:15:26.353991 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5e73ae95c73945f081c298f49d47c904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e73ae95c73945f081c298f49d47c904 already in progress: creating tablet
I20251023 20:15:26.354387 23787 raft_consensus.cc:359] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.354550 23787 raft_consensus.cc:385] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.354598 23787 raft_consensus.cc:740] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.354954 23787 consensus_queue.cc:260] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.355626 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f908f086946949829cac15418a584730 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f908f086946949829cac15418a584730 already in progress: creating tablet
W20251023 20:15:26.356159 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1ae910ac41fd4e349f159ab27e44375a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ae910ac41fd4e349f159ab27e44375a already in progress: creating tablet
I20251023 20:15:26.354789 23786 tablet_bootstrap.cc:654] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.357210 24259 raft_consensus.cc:493] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.357344 24259 raft_consensus.cc:515] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.357965 24259 leader_election.cc:290] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.358525 23787 ts_tablet_manager.cc:1434] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:26.356509 24349 tablet_replica.cc:333] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:26.358963 24349 raft_consensus.cc:2243] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.359375 24349 raft_consensus.cc:2272] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.359400 23787 tablet_bootstrap.cc:492] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.359946 24349 ts_tablet_manager.cc:1916] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.363894 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244fa327fad74aad9e57a7fbcbf908bb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:26.367787 23203 consensus_peers.cc:597] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a -> Peer ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509): Couldn't send request to peer ae5a310b4a514c0f812bda133c756062. Status: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251023 20:15:26.377718 23787 tablet_bootstrap.cc:654] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.380316 23786 tablet_bootstrap.cc:492] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.380535 23786 ts_tablet_manager.cc:1403] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20251023 20:15:26.381325 23786 raft_consensus.cc:359] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.381575 23786 raft_consensus.cc:385] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.381695 23786 raft_consensus.cc:740] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.381964 23786 consensus_queue.cc:260] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.382412 24297 leader_election.cc:290] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.371601 24495 raft_consensus.cc:515] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.373723 23790 tablet_bootstrap.cc:492] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:26.377592 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 45009.6us
I20251023 20:15:26.382792 23790 ts_tablet_manager.cc:1403] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20251023 20:15:26.383108 23786 ts_tablet_manager.cc:1434] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:26.383121 24495 leader_election.cc:290] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:26.383262 23333 consensus_peers.cc:597] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:26.383391 23786 tablet_bootstrap.cc:492] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.384621 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95574277dde44983bb0a43d2b5ad01d5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.384883 23530 raft_consensus.cc:3060] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.385887 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.386408 23790 raft_consensus.cc:359] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.386584 23790 raft_consensus.cc:385] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.386687 23790 raft_consensus.cc:740] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.386932 23790 consensus_queue.cc:260] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.388243 23790 ts_tablet_manager.cc:1434] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:26.388986 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:26.389215 23668 raft_consensus.cc:3060] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.389696 23530 raft_consensus.cc:2468] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.391012 23786 tablet_bootstrap.cc:654] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.391566 24230 raft_consensus.cc:493] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.391721 23668 raft_consensus.cc:2468] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.391738 24230 raft_consensus.cc:515] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.392333 24230 leader_election.cc:290] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.392632 23790 tablet_bootstrap.cc:492] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.392820 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5346a221c78342359eb6e418bd12cd24" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:26.392587 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5346a221c78342359eb6e418bd12cd24" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.393008 23668 raft_consensus.cc:3060] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.394801 23790 tablet_bootstrap.cc:654] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.395498 23668 raft_consensus.cc:2468] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.294049 24363 ts_tablet_manager.cc:1950] T 33b5efb30b5c491c8eafe3ac25688843 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.402727 24349 ts_tablet_manager.cc:1929] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.402885 24349 log.cc:1199] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/7301d6121c9d4c8dbc642e895aa4e9f6
I20251023 20:15:26.403266 24349 ts_tablet_manager.cc:1950] T 7301d6121c9d4c8dbc642e895aa4e9f6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:26.403545 23333 leader_election.cc:343] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:26.403853 23333 leader_election.cc:304] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.404138 24297 raft_consensus.cc:2804] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.404227 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 7301d6121c9d4c8dbc642e895aa4e9f6 (table default.Table_39094702f1ba4174a5eb05a40213e745 [id=2146c6f03d014ccdbc874c613400a8e5]) successfully deleted
I20251023 20:15:26.404274 24297 raft_consensus.cc:697] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.404559 24297 consensus_queue.cc:237] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.407872 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 33b5efb30b5c491c8eafe3ac25688843 (table default.Table_6d625b139835401587825737c53000e3 [id=2ede045acf474fe9a77d38eaed85a15a]) successfully deleted
I20251023 20:15:26.410938 23787 tablet_bootstrap.cc:492] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:26.411021 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 46f700ea3db04a4bad539c943e850a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f700ea3db04a4bad539c943e850a44 already in progress: creating tablet
I20251023 20:15:26.411115 23787 ts_tablet_manager.cc:1403] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20251023 20:15:26.411126 24259 raft_consensus.cc:493] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.411268 24259 raft_consensus.cc:515] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:26.411832 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 4beebee06e5341a29c65d0c3cf985ef4 because the tablet was not found. No further retry: Not found: Tablet not found: 4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:26.411899 24259 leader_election.cc:290] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.411912 23787 raft_consensus.cc:359] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.412101 23787 raft_consensus.cc:385] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.412192 23787 raft_consensus.cc:740] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.412465 23787 consensus_queue.cc:260] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.413022 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 8317a54a5f7b4afe8da9d336268efe43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8317a54a5f7b4afe8da9d336268efe43 already in progress: creating tablet
I20251023 20:15:26.413367 24600 raft_consensus.cc:493] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.413641 24600 raft_consensus.cc:515] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.413864 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
W20251023 20:15:26.414091 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
I20251023 20:15:26.414682 23790 tablet_bootstrap.cc:492] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.414829 23787 ts_tablet_manager.cc:1434] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:26.415263 24349 tablet_replica.cc:333] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:26.415462 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a8ba7ab28dfa4ac38e0b58ba2115495c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8ba7ab28dfa4ac38e0b58ba2115495c already in progress: creating tablet
I20251023 20:15:26.415477 24349 raft_consensus.cc:2243] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.415676 24349 raft_consensus.cc:2272] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.415198 24600 leader_election.cc:290] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.416235 24349 ts_tablet_manager.cc:1916] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.416317 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d951134f6773410386bc57f7f90310c3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:26.417026 23604 leader_election.cc:343] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.421386 23470 leader_election.cc:343] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e0f8d8184a68453abd2792ccc7740aa1
I20251023 20:15:26.421929 23790 ts_tablet_manager.cc:1403] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251023 20:15:26.422103 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d951134f6773410386bc57f7f90310c3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:26.422734 23605 leader_election.cc:343] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.422871 23605 leader_election.cc:304] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.423350 24600 raft_consensus.cc:2749] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.424470 23790 raft_consensus.cc:359] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.424742 23790 raft_consensus.cc:385] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.424819 23790 raft_consensus.cc:740] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.425071 23790 consensus_queue.cc:260] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.425602 23790 ts_tablet_manager.cc:1434] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:26.425947 23790 tablet_bootstrap.cc:492] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.427663 23790 tablet_bootstrap.cc:654] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.427804 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:26.427989 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:26.428125 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:26.428341 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:26.428779 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:26.429535 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:26.429844 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:26.430230 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:26.430655 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:26.430855 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:26.431074 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:26.431329 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:26.431547 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:26.431840 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:26.432626 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:26.432878 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:26.433283 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:26.433487 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:26.433651 23786 tablet_bootstrap.cc:492] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.433830 23786 ts_tablet_manager.cc:1403] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20251023 20:15:26.433861 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:26.434401 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:26.434628 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:26.434870 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:26.435406 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:26.435731 23786 raft_consensus.cc:359] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.435940 23786 raft_consensus.cc:385] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.436021 23786 raft_consensus.cc:740] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.436189 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:26.436270 23786 consensus_queue.cc:260] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.438115 23786 ts_tablet_manager.cc:1434] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:26.439061 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:26.439270 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:26.439646 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]). Expected version 2 got 0
I20251023 20:15:26.439826 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]). Expected version 2 got 0
I20251023 20:15:26.440088 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:26.440330 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:26.440745 23214 catalog_manager.cc:5649] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:26.441242 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
I20251023 20:15:26.441496 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:26.441705 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:26.441920 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:26.442155 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]). Expected version 1 got 0
I20251023 20:15:26.442683 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]). Expected version 1 got 0
I20251023 20:15:26.443728 23786 tablet_bootstrap.cc:492] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.443758 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:26.443061 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]). Expected version 3 got 0
I20251023 20:15:26.445937 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]). Expected version 4 got 0
I20251023 20:15:26.446412 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]). Expected version 2 got 0
I20251023 20:15:26.447125 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:26.447551 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971]). Expected version 1 got 0
I20251023 20:15:26.448024 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:26.448361 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:26.448740 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:26.449070 23214 catalog_manager.cc:5649] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:26.449510 23786 tablet_bootstrap.cc:654] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.449594 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:26.449854 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]). Expected version 3 got 0
I20251023 20:15:26.450125 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:26.450595 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:26.450824 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:26.451009 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:26.451586 23213 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 1e19859a65a144598e1612be89daa3a1
I20251023 20:15:26.452171 23213 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 39) took 1289 ms. Trace:
I20251023 20:15:26.452358 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.452682 24349 ts_tablet_manager.cc:1929] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.452819 24349 log.cc:1199] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/9fc7e18054c440098dc2470143ea7fae
I20251023 20:15:26.452286 23213 rpcz_store.cc:276] 1023 20:15:25.162384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.358775 (+196391us) service_pool.cc:225] Handling call
1023 20:15:25.420645 (+ 61870us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0603796 00007f8997108aa6 00007f89971087d5 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:25.420996 (+ 351us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:25.421132 (+ 136us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:25.421203 (+ 71us) catalog_manager.cc:2824] Locking tablets
1023 20:15:25.569282 (+148079us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:25.947785 (+378503us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:25.947893 (+ 108us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:26.451508 (+503615us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:26.452130 (+ 622us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:25.934074 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:25.934171 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:25.934177 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:25.934179 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:25.934276 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:25.934286 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:25.934290 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:25.934384 (+ 94us) tablet.cc:699] Row locks acquired
1023 20:15:25.934387 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:25.934418 (+ 31us) write_op.cc:270] Start()
1023 20:15:25.934447 (+ 29us) write_op.cc:276] Timestamp: P: 1761250525934368 usec, L: 0
1023 20:15:25.934450 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:25.934594 (+ 144us) log.cc:844] Serialized 734 byte log entry
1023 20:15:25.946279 (+ 11685us) op_driver.cc:464] REPLICATION: finished
1023 20:15:25.947011 (+ 732us) write_op.cc:301] APPLY: starting
1023 20:15:25.947021 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:25.947055 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:25.947086 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:25.947138 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:25.947164 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:25.947171 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:25.947219 (+ 48us) log.cc:844] Serialized 29 byte log entry
1023 20:15:25.947254 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.947266 (+ 12us) write_op.cc:454] Released schema lock
1023 20:15:25.947271 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:25.947324 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.947328 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":141817728,"child_traces":[["op",{"apply.queue_time_us":702,"apply.run_cpu_time_us":307,"apply.run_wall_time_us":307,"num_ops":2,"prepare.queue_time_us":364165,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":598,"raft.queue_time_us":62,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":11813}]]}
I20251023 20:15:26.453208 23213 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_a29ebb47860443d89041e72a86ac92fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:26.453775 23213 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a29ebb47860443d89041e72a86ac92fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:26.454197 24349 ts_tablet_manager.cc:1950] T 9fc7e18054c440098dc2470143ea7fae P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.454744 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 9fc7e18054c440098dc2470143ea7fae (table default.Table_265f006bfcc2450f964410c799f035f9 [id=f0872a169a604701b06c246f1af67ac0]) successfully deleted
W20251023 20:15:26.454916 23215 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d801ca6ce63440abefe0c694889fa9e consensus state has no leader
I20251023 20:15:26.455063 23215 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 42) took 1292 ms. Trace:
I20251023 20:15:26.455153 23215 rpcz_store.cc:276] 1023 20:15:25.162652 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.825943 (+663291us) service_pool.cc:225] Handling call
1023 20:15:25.826291 (+ 348us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:25.826512 (+ 221us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:25.826711 (+ 199us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:25.826792 (+ 81us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:25.947932 (+121140us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:26.455033 (+507101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:25.934704 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:25.934787 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:25.934792 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:25.934793 (+ 1us) tablet.cc:660] Decoding operations
1023 20:15:25.934862 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:25.934869 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:25.934874 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:25.934915 (+ 41us) tablet.cc:699] Row locks acquired
1023 20:15:25.934917 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:25.934941 (+ 24us) write_op.cc:270] Start()
1023 20:15:25.934964 (+ 23us) write_op.cc:276] Timestamp: P: 1761250525934894 usec, L: 0
1023 20:15:25.934967 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:25.935087 (+ 120us) log.cc:844] Serialized 405 byte log entry
1023 20:15:25.946324 (+ 11237us) op_driver.cc:464] REPLICATION: finished
1023 20:15:25.947381 (+ 1057us) write_op.cc:301] APPLY: starting
1023 20:15:25.947391 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:25.947416 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:25.947436 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:25.947462 (+ 26us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:25.947482 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:25.947489 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:25.947528 (+ 39us) log.cc:844] Serialized 23 byte log entry
1023 20:15:25.947565 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.947578 (+ 13us) write_op.cc:454] Released schema lock
1023 20:15:25.947583 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1028,"apply.run_cpu_time_us":252,"apply.run_wall_time_us":252,"num_ops":1,"prepare.queue_time_us":107448,"prepare.run_cpu_time_us":460,"prepare.run_wall_time_us":460,"raft.queue_time_us":35,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":6,"replication_time_us":11343,"spinlock_wait_cycles":1408,"thread_start_us":136426,"threads_started":1}]]}
W20251023 20:15:26.455703 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
I20251023 20:15:26.455865 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_1e09690c826a4c028e998750992ec3d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:26.456358 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e09690c826a4c028e998750992ec3d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:26.456795 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 46f700ea3db04a4bad539c943e850a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f700ea3db04a4bad539c943e850a44 already in progress: creating tablet
W20251023 20:15:26.457268 23210 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a74b3f22b854c1ab2b6cc817b84cf47 consensus state has no leader
W20251023 20:15:26.457338 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
I20251023 20:15:26.457418 23210 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 41) took 1294 ms. Trace:
I20251023 20:15:26.457595 24349 tablet_replica.cc:333] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:26.457973 24349 raft_consensus.cc:2243] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.457532 23210 rpcz_store.cc:276] 1023 20:15:25.162578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.532452 (+369874us) service_pool.cc:225] Handling call
1023 20:15:25.536939 (+ 4487us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0604a36 00007f899710a766 00007f899710a495 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:25.537195 (+ 256us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:25.537416 (+ 221us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:25.537612 (+ 196us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:25.537683 (+ 71us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:25.947697 (+410014us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:26.457381 (+509684us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:25.933295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:25.933421 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:25.933429 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:25.933431 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:25.933539 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:25.933550 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:25.933556 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:25.933636 (+ 80us) tablet.cc:699] Row locks acquired
1023 20:15:25.933638 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:25.933683 (+ 45us) write_op.cc:270] Start()
1023 20:15:25.933723 (+ 40us) write_op.cc:276] Timestamp: P: 1761250525933629 usec, L: 0
1023 20:15:25.933727 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:25.933953 (+ 226us) log.cc:844] Serialized 405 byte log entry
1023 20:15:25.946212 (+ 12259us) op_driver.cc:464] REPLICATION: finished
1023 20:15:25.946532 (+ 320us) write_op.cc:301] APPLY: starting
1023 20:15:25.946549 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:25.946590 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:25.946617 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:25.946669 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:25.946701 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:25.946709 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:25.946784 (+ 75us) log.cc:844] Serialized 23 byte log entry
1023 20:15:25.946851 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.946869 (+ 18us) write_op.cc:454] Released schema lock
1023 20:15:25.946876 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:25.946942 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.946946 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":9825536,"child_traces":[["op",{"apply.queue_time_us":269,"apply.run_cpu_time_us":403,"apply.run_wall_time_us":403,"num_ops":1,"prepare.queue_time_us":395154,"prepare.run_cpu_time_us":759,"prepare.run_wall_time_us":758,"raft.queue_time_us":655,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":12455}]]}
I20251023 20:15:26.458302 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_b3b6a3c4f36f4f888a6b5c483002e6ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:26.458766 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3b6a3c4f36f4f888a6b5c483002e6ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:26.459772 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:26.464011 24349 raft_consensus.cc:2272] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.464610 24349 ts_tablet_manager.cc:1916] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.469460 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:26.470564 23213 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 46) took 1280 ms. Trace:
I20251023 20:15:26.474880 23213 rpcz_store.cc:276] 1023 20:15:25.190093 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.452977 (+1262884us) service_pool.cc:225] Handling call
1023 20:15:26.453871 (+ 894us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:26.454302 (+ 431us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:26.470314 (+ 16012us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:26.470423 (+ 109us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:26.470520 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.458387 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.458476 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.458482 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:26.458484 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:26.458559 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.458569 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.458573 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.458657 (+ 84us) tablet.cc:699] Row locks acquired
1023 20:15:26.458659 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:26.458692 (+ 33us) write_op.cc:270] Start()
1023 20:15:26.458723 (+ 31us) write_op.cc:276] Timestamp: P: 1761250526458644 usec, L: 0
1023 20:15:26.458726 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.458876 (+ 150us) log.cc:844] Serialized 433 byte log entry
1023 20:15:26.465976 (+ 7100us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.466112 (+ 136us) write_op.cc:301] APPLY: starting
1023 20:15:26.466127 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.466171 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.466196 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.466301 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.466334 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:26.466341 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:26.466411 (+ 70us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.466475 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.466492 (+ 17us) write_op.cc:454] Released schema lock
1023 20:15:26.466499 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.466568 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.466572 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":448,"apply.run_wall_time_us":447,"num_ops":2,"prepare.queue_time_us":3680,"prepare.run_cpu_time_us":575,"prepare.run_wall_time_us":574,"raft.queue_time_us":59,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":7222}]]}
I20251023 20:15:26.476511 24600 raft_consensus.cc:493] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.476692 24600 raft_consensus.cc:515] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.477353 24600 leader_election.cc:290] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.478382 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.478631 23399 raft_consensus.cc:2468] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:26.479254 23605 leader_election.cc:304] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.479842 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:26.480505 23604 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.480938 24553 raft_consensus.cc:2804] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.481047 24553 raft_consensus.cc:493] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.481153 24553 raft_consensus.cc:3060] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.483531 24349 ts_tablet_manager.cc:1929] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:26.483657 24349 log.cc:1199] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:26.483706 24553 raft_consensus.cc:515] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.484310 24553 leader_election.cc:290] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.484479 24553 raft_consensus.cc:493] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.484599 24553 raft_consensus.cc:515] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.485219 24553 leader_election.cc:290] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:26.486929 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb915d114fc0421999d079c23cad8eb5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.487146 23397 raft_consensus.cc:2468] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:26.487548 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.487735 23399 raft_consensus.cc:3060] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.490552 23399 raft_consensus.cc:2468] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:26.491577 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb915d114fc0421999d079c23cad8eb5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:26.492213 23604 leader_election.cc:343] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.492660 23605 leader_election.cc:304] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.493116 23605 leader_election.cc:304] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.493347 24160 raft_consensus.cc:2804] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.493434 24160 raft_consensus.cc:493] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.493502 24160 raft_consensus.cc:3060] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.495976 24160 raft_consensus.cc:515] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.496579 24160 leader_election.cc:290] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:26.496766 24160 raft_consensus.cc:2804] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.497279 24349 ts_tablet_manager.cc:1950] T 4beebee06e5341a29c65d0c3cf985ef4 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.497874 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 4beebee06e5341a29c65d0c3cf985ef4 (table default.Table_3220c87e07554676ba34368830409e89 [id=85bda6747ffe49f1b5a123f41013c426]) successfully deleted
I20251023 20:15:26.498332 24160 raft_consensus.cc:697] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
W20251023 20:15:26.501147 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 8317a54a5f7b4afe8da9d336268efe43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8317a54a5f7b4afe8da9d336268efe43 already in progress: creating tablet
W20251023 20:15:26.502259 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
I20251023 20:15:26.502295 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb915d114fc0421999d079c23cad8eb5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.473034 23215 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 47) took 1282 ms. Trace:
I20251023 20:15:26.503403 23215 rpcz_store.cc:276] 1023 20:15:25.190193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.455680 (+1265487us) service_pool.cc:225] Handling call
1023 20:15:26.456442 (+ 762us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:26.456876 (+ 434us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:26.472824 (+ 15948us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:26.472918 (+ 94us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:26.472991 (+ 73us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.463493 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.463620 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.463625 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:26.463627 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:26.463699 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.463707 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.463711 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.463770 (+ 59us) tablet.cc:699] Row locks acquired
1023 20:15:26.463772 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:26.463798 (+ 26us) write_op.cc:270] Start()
1023 20:15:26.463822 (+ 24us) write_op.cc:276] Timestamp: P: 1761250526463752 usec, L: 0
1023 20:15:26.463825 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.463949 (+ 124us) log.cc:844] Serialized 433 byte log entry
1023 20:15:26.471538 (+ 7589us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.472410 (+ 872us) write_op.cc:301] APPLY: starting
1023 20:15:26.472421 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.472452 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.472480 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.472568 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.472595 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:26.472602 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:26.472672 (+ 70us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.472714 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.472729 (+ 15us) write_op.cc:454] Released schema lock
1023 20:15:26.472735 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":839,"apply.run_cpu_time_us":378,"apply.run_wall_time_us":378,"num_ops":2,"prepare.queue_time_us":806,"prepare.run_cpu_time_us":529,"prepare.run_wall_time_us":529,"raft.queue_time_us":276,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":7695,"threads_started":1}]]}
I20251023 20:15:26.472641 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 48) took 1264 ms. Trace:
I20251023 20:15:26.504612 24349 tablet_replica.cc:333] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:26.504819 24349 raft_consensus.cc:2243] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.504998 24349 raft_consensus.cc:2272] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.504606 23210 rpcz_store.cc:276] 1023 20:15:25.207751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.458135 (+1250384us) service_pool.cc:225] Handling call
1023 20:15:26.458845 (+ 710us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:26.459238 (+ 393us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:26.472348 (+ 13110us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:26.472444 (+ 96us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:26.472528 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.462890 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.462985 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.462991 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:26.462994 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:26.463070 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.463080 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.463085 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.463158 (+ 73us) tablet.cc:699] Row locks acquired
1023 20:15:26.463160 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:26.463191 (+ 31us) write_op.cc:270] Start()
1023 20:15:26.463218 (+ 27us) write_op.cc:276] Timestamp: P: 1761250526463143 usec, L: 0
1023 20:15:26.463221 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.463378 (+ 157us) log.cc:844] Serialized 433 byte log entry
1023 20:15:26.471473 (+ 8095us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.471871 (+ 398us) write_op.cc:301] APPLY: starting
1023 20:15:26.471887 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.471927 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.471959 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.472064 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.472095 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:26.472103 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:26.472173 (+ 70us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.472230 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.472246 (+ 16us) write_op.cc:454] Released schema lock
1023 20:15:26.472253 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.472340 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.472344 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":343,"apply.run_cpu_time_us":450,"apply.run_wall_time_us":449,"num_ops":2,"prepare.queue_time_us":3263,"prepare.run_cpu_time_us":574,"prepare.run_wall_time_us":574,"raft.queue_time_us":795,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":8224}]]}
I20251023 20:15:26.505326 23210 master_service.cc:954] ReplaceTablet: received request to replace tablet a8ba7ab28dfa4ac38e0b58ba2115495c from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:26.506126 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.506819 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb915d114fc0421999d079c23cad8eb5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.507081 23399 raft_consensus.cc:3060] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.508560 24160 consensus_queue.cc:237] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.508795 23604 leader_election.cc:343] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.509305 23604 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.510274 23399 raft_consensus.cc:2468] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:26.510879 23605 leader_election.cc:304] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.511162 24349 ts_tablet_manager.cc:1916] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.511452 24553 raft_consensus.cc:2804] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.511616 24553 raft_consensus.cc:697] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:26.511873 24553 consensus_queue.cc:237] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.518332 23790 tablet_bootstrap.cc:492] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.518499 23790 ts_tablet_manager.cc:1403] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20251023 20:15:26.519279 23790 raft_consensus.cc:359] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.519366 23333 leader_election.cc:304] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.519487 23790 raft_consensus.cc:385] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.519584 23790 raft_consensus.cc:740] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:26.519665 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 109848us
I20251023 20:15:26.520331 23786 tablet_bootstrap.cc:492] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.520304 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244fa327fad74aad9e57a7fbcbf908bb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.520470 23786 ts_tablet_manager.cc:1403] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.000s
I20251023 20:15:26.521009 24297 raft_consensus.cc:2804] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.521749 23786 raft_consensus.cc:359] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.521942 23786 raft_consensus.cc:385] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.522040 23786 raft_consensus.cc:740] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.522248 23786 consensus_queue.cc:260] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:26.522948 23471 leader_election.cc:343] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: e0f8d8184a68453abd2792ccc7740aa1
I20251023 20:15:26.522809 24297 raft_consensus.cc:697] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.522992 23786 ts_tablet_manager.cc:1434] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:26.523065 23471 leader_election.cc:304] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:26.524358 23471 leader_election.cc:304] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:26.524417 23786 tablet_bootstrap.cc:492] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.524959 24615 raft_consensus.cc:2749] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.525147 24615 raft_consensus.cc:2749] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.525115 23790 consensus_queue.cc:260] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.523243 24297 consensus_queue.cc:237] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.528147 23790 ts_tablet_manager.cc:1434] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:26.528203 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.528560 23668 raft_consensus.cc:2468] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.533386 24607 tablet.cc:1722] T c317dc31cb0448d985512e6ba98735be P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:26.533926 23790 tablet_bootstrap.cc:492] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.535656 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:26.537165 24619 tablet.cc:1722] T c317dc31cb0448d985512e6ba98735be P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:26.538403 23213 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_8e0012945b9e401793b99da25bc5d76c" } modify_external_catalogs: true
I20251023 20:15:26.538688 23213 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_8e0012945b9e401793b99da25bc5d76c" } modify_external_catalogs: true
I20251023 20:15:26.539496 23210 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 51) took 1066 ms. Trace:
I20251023 20:15:26.539580 23210 rpcz_store.cc:276] 1023 20:15:25.473423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.505260 (+1031837us) service_pool.cc:225] Handling call
1023 20:15:26.539210 (+ 33950us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a060f4eb 00007f8997113727 00007f8997113455 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:26.539436 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74797312}
I20251023 20:15:26.539958 23210 master_service.cc:954] ReplaceTablet: received request to replace tablet 9a4f012469fa4e8e827e485287ad76dd from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:26.540575 23790 tablet_bootstrap.cc:654] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.540536 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_8587355637694585afe381e3e77c3958"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:26.541391 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8587355637694585afe381e3e77c3958 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:26.544417 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.547611 23786 tablet_bootstrap.cc:654] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.552058 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:26.552453 24349 ts_tablet_manager.cc:1929] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.552567 24349 log.cc:1199] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/7540354f8577482785e6341fa9022315
I20251023 20:15:26.552951 24349 ts_tablet_manager.cc:1950] T 7540354f8577482785e6341fa9022315 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.554061 24349 tablet_replica.cc:333] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:26.554262 24349 raft_consensus.cc:2243] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.554409 24349 raft_consensus.cc:2272] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.554921 24349 ts_tablet_manager.cc:1916] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.557502 23215 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 50) took 1265 ms. Trace:
I20251023 20:15:26.557627 23215 rpcz_store.cc:276] 1023 20:15:25.291612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.504041 (+1212429us) service_pool.cc:225] Handling call
1023 20:15:26.540486 (+ 36445us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a06027a3 00007f8997106de6 00007f8997106b15 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:26.541482 (+ 996us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:26.541988 (+ 506us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:26.557266 (+ 15278us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:26.557359 (+ 93us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:26.557454 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.542669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.542761 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.542768 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:26.542770 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:26.542855 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.542866 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.542871 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.542946 (+ 75us) tablet.cc:699] Row locks acquired
1023 20:15:26.542949 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:26.542984 (+ 35us) write_op.cc:270] Start()
1023 20:15:26.543014 (+ 30us) write_op.cc:276] Timestamp: P: 1761250526542966 usec, L: 0
1023 20:15:26.543018 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.543167 (+ 149us) log.cc:844] Serialized 433 byte log entry
1023 20:15:26.548368 (+ 5201us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.549255 (+ 887us) write_op.cc:301] APPLY: starting
1023 20:15:26.549264 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.549291 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.549314 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.549401 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.549428 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:26.549435 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:26.549490 (+ 55us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.549530 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.549545 (+ 15us) write_op.cc:454] Released schema lock
1023 20:15:26.549550 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.549618 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.549622 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":83173248,"child_traces":[["op",{"apply.queue_time_us":852,"apply.run_cpu_time_us":344,"apply.run_wall_time_us":344,"num_ops":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":583,"raft.queue_time_us":325,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":5325}]]}
I20251023 20:15:26.558292 23215 master_service.cc:954] ReplaceTablet: received request to replace tablet e5368813164949ff80e27bf59dcc931a from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:26.559563 23210 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 52) took 1078 ms. Trace:
I20251023 20:15:26.559670 23210 rpcz_store.cc:276] 1023 20:15:25.480620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.539874 (+1059254us) service_pool.cc:225] Handling call
1023 20:15:26.559520 (+ 19646us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:26.560003 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_9c045162e1824edbbcf0718c49e3b185"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:26.560534 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c045162e1824edbbcf0718c49e3b185 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:26.563056 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 7540354f8577482785e6341fa9022315 (table default.Table_410a6baf06a642b18af37bebddac662f [id=75026c75215e419983355275cf38f7f3]) successfully deleted
W20251023 20:15:26.563299 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
W20251023 20:15:26.563513 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
I20251023 20:15:26.564497 23646 tablet_service.cc:1505] Processing CreateTablet for tablet bc7793d1370d47ac8ee66d0246aba8fe (DEFAULT_TABLE table=default.Table_94f35966189741de98430b366fccb99f [id=e08e2bb1c801450f99912c35dc0371e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.565547 23215 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet e5368813164949ff80e27bf59dcc931a
I20251023 20:15:26.566000 23215 catalog_manager.cc:6719] ReplaceTablet: tablet e5368813164949ff80e27bf59dcc931a deleted and replaced by tablet abdad7d76eb8441496d38642382f74bb
I20251023 20:15:26.566115 23215 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 53) took 1063 ms. Trace:
I20251023 20:15:26.566228 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet e5368813164949ff80e27bf59dcc931a with delete_type TABLET_DATA_DELETED (replaced by tablet abdad7d76eb8441496d38642382f74bb) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:26.566190 23215 rpcz_store.cc:276] 1023 20:15:25.502467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.558223 (+1055756us) service_pool.cc:225] Handling call
1023 20:15:26.566068 (+ 7845us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.559316 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.559403 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.559409 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:26.559412 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:26.559502 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.559510 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.559514 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.559581 (+ 67us) tablet.cc:699] Row locks acquired
1023 20:15:26.559583 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:26.559624 (+ 41us) write_op.cc:270] Start()
1023 20:15:26.559652 (+ 28us) write_op.cc:276] Timestamp: P: 1761250526559597 usec, L: 0
1023 20:15:26.559656 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.559806 (+ 150us) log.cc:844] Serialized 665 byte log entry
1023 20:15:26.564219 (+ 4413us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.564632 (+ 413us) write_op.cc:301] APPLY: starting
1023 20:15:26.564646 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.564685 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.564725 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.564825 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.564881 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:26.564889 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:26.564960 (+ 71us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.565017 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.565036 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:26.565042 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.565125 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.565128 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":348,"apply.run_cpu_time_us":490,"apply.run_wall_time_us":500,"num_ops":2,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":579,"prepare.run_wall_time_us":580,"raft.queue_time_us":53,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":4539}]]}
I20251023 20:15:26.572968 23502 tablet_service.cc:1505] Processing CreateTablet for tablet bc7793d1370d47ac8ee66d0246aba8fe (DEFAULT_TABLE table=default.Table_94f35966189741de98430b366fccb99f [id=e08e2bb1c801450f99912c35dc0371e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.573477 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc7793d1370d47ac8ee66d0246aba8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.576164 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 54) took 1007 ms. Trace:
I20251023 20:15:26.576296 23210 rpcz_store.cc:276] 1023 20:15:25.568325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.559821 (+991496us) service_pool.cc:225] Handling call
1023 20:15:26.560653 (+ 832us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:26.561062 (+ 409us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:26.575928 (+ 14866us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:26.576028 (+ 100us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:26.576122 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.571122 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.571242 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.571247 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:26.571250 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:26.571337 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.571348 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.571353 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.571424 (+ 71us) tablet.cc:699] Row locks acquired
1023 20:15:26.571427 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:26.571459 (+ 32us) write_op.cc:270] Start()
1023 20:15:26.571489 (+ 30us) write_op.cc:276] Timestamp: P: 1761250526571439 usec, L: 0
1023 20:15:26.571492 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.571661 (+ 169us) log.cc:844] Serialized 433 byte log entry
1023 20:15:26.574763 (+ 3102us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.575243 (+ 480us) write_op.cc:301] APPLY: starting
1023 20:15:26.575259 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.575301 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.575324 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.575432 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.575462 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:26.575469 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:26.575536 (+ 67us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.575589 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.575616 (+ 27us) write_op.cc:454] Released schema lock
1023 20:15:26.575622 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.575689 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.575693 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":418,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":445,"num_ops":2,"prepare.queue_time_us":9646,"prepare.run_cpu_time_us":639,"prepare.run_wall_time_us":638,"raft.queue_time_us":138,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":189,"replication_time_us":3244}]]}
I20251023 20:15:26.576989 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet e5368813164949ff80e27bf59dcc931a with delete_type TABLET_DATA_DELETED (replaced by tablet abdad7d76eb8441496d38642382f74bb) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:26.577000 23210 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_ede581d1b2824c17933809e745e8ab07" } modify_external_catalogs: true
I20251023 20:15:26.577723 23210 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_ede581d1b2824c17933809e745e8ab07" } modify_external_catalogs: true
I20251023 20:15:26.577865 23790 tablet_bootstrap.cc:492] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.578058 23790 ts_tablet_manager.cc:1403] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.001s
I20251023 20:15:26.582352 23790 raft_consensus.cc:359] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.582566 23790 raft_consensus.cc:385] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.582644 23790 raft_consensus.cc:740] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.582861 23790 consensus_queue.cc:260] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.583447 23786 tablet_bootstrap.cc:492] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.583613 23786 ts_tablet_manager.cc:1403] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20251023 20:15:26.583987 23790 ts_tablet_manager.cc:1434] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:26.584359 23786 raft_consensus.cc:359] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.584544 23786 raft_consensus.cc:385] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.584599 23790 tablet_bootstrap.cc:492] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.584614 23786 raft_consensus.cc:740] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.584836 23786 consensus_queue.cc:260] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.586687 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 0140a384382941a68e9a918d4a82a449 (DEFAULT_TABLE table=default.Table_03ddad2769a4450b81d0d8028293a0b2 [id=20d0cb227c7242b29c5ee5423268da12]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:26.586843 23333 leader_election.cc:343] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.587108 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0140a384382941a68e9a918d4a82a449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.587167 23333 leader_election.cc:304] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:26.587563 23333 leader_election.cc:343] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.413380 24363 tablet_replica.cc:333] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.593673 23787 tablet_bootstrap.cc:492] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.595186 24363 raft_consensus.cc:2243] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:26.595384 23787 tablet_bootstrap.cc:654] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.595360 24259 raft_consensus.cc:493] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.595489 24259 raft_consensus.cc:515] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:26.595566 24363 raft_consensus.cc:2272] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.596138 24259 leader_election.cc:290] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.596154 24363 ts_tablet_manager.cc:1916] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.596429 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc7793d1370d47ac8ee66d0246aba8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.597522 24230 raft_consensus.cc:2804] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.597673 24230 raft_consensus.cc:697] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.597648 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 878508d6bf0d4a59b6f64155361b4418 (DEFAULT_TABLE table=default.Table_3abcf10252074989b84e5566c762a99d [id=ba74b6d39572458e9f01923f709c2d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.598034 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878508d6bf0d4a59b6f64155361b4418. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.597960 24230 consensus_queue.cc:237] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.600062 23634 tablet_service.cc:1505] Processing CreateTablet for tablet 0140a384382941a68e9a918d4a82a449 (DEFAULT_TABLE table=default.Table_03ddad2769a4450b81d0d8028293a0b2 [id=20d0cb227c7242b29c5ee5423268da12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.600492 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0140a384382941a68e9a918d4a82a449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.601364 23510 tablet_service.cc:1505] Processing CreateTablet for tablet 2546fa8cb2004463af57f29ba28fdc1d (DEFAULT_TABLE table=default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0 [id=b9cb00bd8954475b909036fd9f123b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.601691 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2546fa8cb2004463af57f29ba28fdc1d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:26.604293 23333 consensus_peers.cc:597] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1a6fe726ef844829a7ffec9d4494637. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:26.604472 23333 leader_election.cc:343] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.605398 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 878508d6bf0d4a59b6f64155361b4418 (DEFAULT_TABLE table=default.Table_3abcf10252074989b84e5566c762a99d [id=ba74b6d39572458e9f01923f709c2d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.605809 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878508d6bf0d4a59b6f64155361b4418. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.606526 23786 ts_tablet_manager.cc:1434] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251023 20:15:26.607403 23786 tablet_bootstrap.cc:492] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.609356 23640 tablet_service.cc:1505] Processing CreateTablet for tablet 2546fa8cb2004463af57f29ba28fdc1d (DEFAULT_TABLE table=default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0 [id=b9cb00bd8954475b909036fd9f123b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.609674 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2546fa8cb2004463af57f29ba28fdc1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.609915 23786 tablet_bootstrap.cc:654] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:26.610054 23205 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1e19859a65a144598e1612be89daa3a1 because the tablet was not found. No further retry: Not found: Tablet not found: 1e19859a65a144598e1612be89daa3a1
I20251023 20:15:26.610935 23333 leader_election.cc:304] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.611333 24495 raft_consensus.cc:2804] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.611443 24495 raft_consensus.cc:493] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.611545 24495 raft_consensus.cc:3060] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.614142 24495 raft_consensus.cc:515] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.614424 24621 raft_consensus.cc:493] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:26.614614 23470 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 585a9af140d9479fb83d13e68a62300c
I20251023 20:15:26.614614 24621 raft_consensus.cc:515] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.614885 24495 leader_election.cc:290] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.615097 24555 tablet.cc:1722] T c317dc31cb0448d985512e6ba98735be P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:26.615195 23362 tablet_service.cc:1505] Processing CreateTablet for tablet bc7793d1370d47ac8ee66d0246aba8fe (DEFAULT_TABLE table=default.Table_94f35966189741de98430b366fccb99f [id=e08e2bb1c801450f99912c35dc0371e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.615252 24621 leader_election.cc:290] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.615561 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc7793d1370d47ac8ee66d0246aba8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.617728 24621 raft_consensus.cc:493] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.618078 24621 raft_consensus.cc:515] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.618606 24621 leader_election.cc:290] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.623035 23637 tablet_service.cc:1505] Processing CreateTablet for tablet e289b51f48a942fe91b95f528130f89b (DEFAULT_TABLE table=default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.623325 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet e5368813164949ff80e27bf59dcc931a with delete_type TABLET_DATA_DELETED (replaced by tablet abdad7d76eb8441496d38642382f74bb) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:26.623412 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e289b51f48a942fe91b95f528130f89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.623951 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 0140a384382941a68e9a918d4a82a449 (DEFAULT_TABLE table=default.Table_03ddad2769a4450b81d0d8028293a0b2 [id=20d0cb227c7242b29c5ee5423268da12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.624269 24636 tablet_replica.cc:333] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:26.624293 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0140a384382941a68e9a918d4a82a449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.624454 24636 raft_consensus.cc:2243] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:26.624864 24636 raft_consensus.cc:2272] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.625439 24636 ts_tablet_manager.cc:1916] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.625994 23787 tablet_bootstrap.cc:492] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.626269 23787 ts_tablet_manager.cc:1403] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.009s
I20251023 20:15:26.627112 23787 raft_consensus.cc:359] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.627285 23787 raft_consensus.cc:385] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.627372 23787 raft_consensus.cc:740] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.627617 23787 consensus_queue.cc:260] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.628137 23787 ts_tablet_manager.cc:1434] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:26.628420 23787 tablet_bootstrap.cc:492] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.629593 24553 raft_consensus.cc:493] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.629724 24553 raft_consensus.cc:515] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.629976 23787 tablet_bootstrap.cc:654] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.630529 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 878508d6bf0d4a59b6f64155361b4418 (DEFAULT_TABLE table=default.Table_3abcf10252074989b84e5566c762a99d [id=ba74b6d39572458e9f01923f709c2d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.630919 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878508d6bf0d4a59b6f64155361b4418. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.630965 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0858358aabff47a69dc77f4669b225c5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.631211 23516 tablet_service.cc:1505] Processing CreateTablet for tablet cc488f0beec14d0eb869b7474ec16205 (DEFAULT_TABLE table=default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.631556 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc488f0beec14d0eb869b7474ec16205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.631907 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0858358aabff47a69dc77f4669b225c5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.632472 24553 leader_election.cc:290] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:26.632750 23604 leader_election.cc:343] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.632987 23605 leader_election.cc:343] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.633111 23605 leader_election.cc:304] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.633484 24553 raft_consensus.cc:2749] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.637157 23362 tablet_service.cc:1505] Processing CreateTablet for tablet cc488f0beec14d0eb869b7474ec16205 (DEFAULT_TABLE table=default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.637658 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc488f0beec14d0eb869b7474ec16205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.637764 23515 tablet_service.cc:1505] Processing CreateTablet for tablet e289b51f48a942fe91b95f528130f89b (DEFAULT_TABLE table=default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.638082 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e289b51f48a942fe91b95f528130f89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.643412 23371 tablet_service.cc:1505] Processing CreateTablet for tablet e289b51f48a942fe91b95f528130f89b (DEFAULT_TABLE table=default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.643812 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e289b51f48a942fe91b95f528130f89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.645463 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 6ecfa16e75f543fab12fa0bac4d9e3e8 (DEFAULT_TABLE table=default.Table_34d8a4d24be94b1ea676b7a0df81b96c [id=7b824b39feed48148896732770542d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.645892 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ecfa16e75f543fab12fa0bac4d9e3e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:26.650485 23471 leader_election.cc:343] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 585a9af140d9479fb83d13e68a62300c
I20251023 20:15:26.650652 23471 leader_election.cc:304] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:26.651039 24259 raft_consensus.cc:2749] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.652315 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.653853 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:26.654057 23668 raft_consensus.cc:3060] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.656699 23668 raft_consensus.cc:2468] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.657963 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "925e53d9ea6f4c7baa38551eb917828d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.637180 24636 ts_tablet_manager.cc:1929] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:26.658272 24636 log.cc:1199] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e5368813164949ff80e27bf59dcc931a
I20251023 20:15:26.658691 24636 ts_tablet_manager.cc:1950] T e5368813164949ff80e27bf59dcc931a P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:26.659137 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 2546fa8cb2004463af57f29ba28fdc1d (DEFAULT_TABLE table=default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0 [id=b9cb00bd8954475b909036fd9f123b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.659802 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "925e53d9ea6f4c7baa38551eb917828d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.660028 23668 raft_consensus.cc:2468] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.661926 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a708cd011802458aa0e2991fd4b4a2d3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.662233 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a708cd011802458aa0e2991fd4b4a2d3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.662433 23668 raft_consensus.cc:2468] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.662000 24621 raft_consensus.cc:493] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.663506 24621 raft_consensus.cc:515] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.664175 24621 leader_election.cc:290] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.664285 23634 tablet_service.cc:1505] Processing CreateTablet for tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3 (DEFAULT_TABLE table=default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.664692 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.666405 23790 tablet_bootstrap.cc:654] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.666842 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2546fa8cb2004463af57f29ba28fdc1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.664292 23646 tablet_service.cc:1505] Processing CreateTablet for tablet cc488f0beec14d0eb869b7474ec16205 (DEFAULT_TABLE table=default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.664984 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 6ecfa16e75f543fab12fa0bac4d9e3e8 (DEFAULT_TABLE table=default.Table_34d8a4d24be94b1ea676b7a0df81b96c [id=7b824b39feed48148896732770542d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.667564 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc488f0beec14d0eb869b7474ec16205. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.674467 24363 ts_tablet_manager.cc:1929] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.674962 23396 raft_consensus.cc:3060] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:26.678232 23333 consensus_peers.cc:597] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7645c36c983f438f8b1223256fd3ad2c. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:26.679644 24621 consensus_queue.cc:1048] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.680501 23333 leader_election.cc:304] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.680869 24621 raft_consensus.cc:2804] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.680992 24621 raft_consensus.cc:697] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.681345 24621 consensus_queue.cc:237] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.682704 23786 tablet_bootstrap.cc:492] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.682873 23786 ts_tablet_manager.cc:1403] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.000s
I20251023 20:15:26.683701 23786 raft_consensus.cc:359] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.683781 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 886838284a0548f6bb089f62c686165e (DEFAULT_TABLE table=default.Table_95fb2c2528f44d8d84f4cd67d9663502 [id=4867e660fb23438495400e364dba7b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.683933 23786 raft_consensus.cc:385] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.684026 23786 raft_consensus.cc:740] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.684188 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886838284a0548f6bb089f62c686165e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.684535 23786 consensus_queue.cc:260] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.685451 23786 ts_tablet_manager.cc:1434] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:26.685840 23333 leader_election.cc:304] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.686347 23786 tablet_bootstrap.cc:492] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.687676 24259 raft_consensus.cc:493] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.687839 24259 raft_consensus.cc:515] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.688493 24259 leader_election.cc:290] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.688832 24230 raft_consensus.cc:2804] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.688995 24230 raft_consensus.cc:493] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.689157 24230 raft_consensus.cc:3060] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.691360 23786 tablet_bootstrap.cc:654] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:26.692188 23470 leader_election.cc:343] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a8ba7ab28dfa4ac38e0b58ba2115495c
I20251023 20:15:26.692344 24230 raft_consensus.cc:515] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.692673 23510 tablet_service.cc:1505] Processing CreateTablet for tablet b74d2e6d360c4200ae8740dddc206401 (DEFAULT_TABLE table=default.Table_2afbf83f519f4b7da2c4acc31ab18380 [id=6736a50ebad44c42985a5be788bfc43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.693123 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74d2e6d360c4200ae8740dddc206401. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.693598 24230 leader_election.cc:290] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.694140 24363 log.cc:1199] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/8289ce37fa074891beb584bf2fff9747
I20251023 20:15:26.693889 23333 leader_election.cc:304] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:26.681104 23646 tablet_service.cc:1505] Processing CreateTablet for tablet 886838284a0548f6bb089f62c686165e (DEFAULT_TABLE table=default.Table_95fb2c2528f44d8d84f4cd67d9663502 [id=4867e660fb23438495400e364dba7b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.682822 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ecfa16e75f543fab12fa0bac4d9e3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.695189 24363 ts_tablet_manager.cc:1950] T 8289ce37fa074891beb584bf2fff9747 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.696408 24495 raft_consensus.cc:2804] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.696517 24495 raft_consensus.cc:493] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.696588 24495 raft_consensus.cc:3060] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.696554 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3 (DEFAULT_TABLE table=default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.696915 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.699163 24495 raft_consensus.cc:515] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.699764 24495 leader_election.cc:290] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.699931 24495 raft_consensus.cc:493] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.700037 24495 raft_consensus.cc:515] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.700637 24495 leader_election.cc:290] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.689343 24553 raft_consensus.cc:493] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.701689 24553 raft_consensus.cc:515] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.702261 23634 tablet_service.cc:1505] Processing CreateTablet for tablet b74d2e6d360c4200ae8740dddc206401 (DEFAULT_TABLE table=default.Table_2afbf83f519f4b7da2c4acc31ab18380 [id=6736a50ebad44c42985a5be788bfc43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.702342 24553 leader_election.cc:290] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.702620 23787 tablet_bootstrap.cc:492] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.702804 23787 ts_tablet_manager.cc:1403] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.018s
I20251023 20:15:26.702867 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.696157 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886838284a0548f6bb089f62c686165e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:26.703459 23605 leader_election.cc:343] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.703735 23787 raft_consensus.cc:359] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.703922 23787 raft_consensus.cc:385] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.703998 23787 raft_consensus.cc:740] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.704238 23787 consensus_queue.cc:260] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.704900 23787 ts_tablet_manager.cc:1434] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:26.705271 23787 tablet_bootstrap.cc:492] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.707561 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74d2e6d360c4200ae8740dddc206401. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.707697 23787 tablet_bootstrap.cc:654] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.711995 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 6ecfa16e75f543fab12fa0bac4d9e3e8 (DEFAULT_TABLE table=default.Table_34d8a4d24be94b1ea676b7a0df81b96c [id=7b824b39feed48148896732770542d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.712410 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ecfa16e75f543fab12fa0bac4d9e3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.713660 23790 tablet_bootstrap.cc:492] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.713819 23790 ts_tablet_manager.cc:1403] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.130s user 0.002s sys 0.001s
I20251023 20:15:26.713846 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 59) took 1105 ms. Trace:
I20251023 20:15:26.713927 24363 rpcz_store.cc:276] 1023 20:15:25.608665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.608850 (+ 185us) service_pool.cc:225] Handling call
1023 20:15:26.413345 (+804495us) ts_tablet_manager.cc:1151] Deleting tablet 8289ce37fa074891beb584bf2fff9747
1023 20:15:26.413352 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.607536 (+194184us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.674359 (+ 66823us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.713791 (+ 39432us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6998,"tablet-delete.queue_time_us":804320}
I20251023 20:15:26.714429 24363 tablet_replica.cc:333] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.714625 24363 raft_consensus.cc:2243] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:26.714679 23790 raft_consensus.cc:359] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.714849 23790 raft_consensus.cc:385] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.714941 23790 raft_consensus.cc:740] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.714955 24363 raft_consensus.cc:2272] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.715219 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 8289ce37fa074891beb584bf2fff9747 (table default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]) successfully deleted
I20251023 20:15:26.715368 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.715502 24363 ts_tablet_manager.cc:1916] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.715552 23790 consensus_queue.cc:260] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.715740 24259 consensus_queue.cc:1048] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:26.716375 23604 leader_election.cc:343] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.716523 23604 leader_election.cc:304] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.717475 24553 raft_consensus.cc:2749] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.723189 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3 (DEFAULT_TABLE table=default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.725919 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 4c0f207603f241669afd4523b0f9f82a (DEFAULT_TABLE table=default.Table_64ad686282f94cb7ac869f59933669f1 [id=91b7c90942ea4aa8bb8e4b32e6d7c5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.726812 23640 tablet_service.cc:1505] Processing CreateTablet for tablet fe49ee5aaf784ef7a9078ff504ddee70 (DEFAULT_TABLE table=default.Table_21f538f7569b437b98fe8a5274592880 [id=6a431dc140754c84af27e9bd3ce01a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.726872 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.727361 23502 tablet_service.cc:1505] Processing CreateTablet for tablet 94b4838644a9403693572350b1bdd0e2 (DEFAULT_TABLE table=default.Table_222ab66f7150462aad2efe8d16484b94 [id=f52ec73cfcfe4b36a08422d69aabf321]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.727999 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 94b4838644a9403693572350b1bdd0e2 (DEFAULT_TABLE table=default.Table_222ab66f7150462aad2efe8d16484b94 [id=f52ec73cfcfe4b36a08422d69aabf321]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.728137 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b4838644a9403693572350b1bdd0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.723953 23790 ts_tablet_manager.cc:1434] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:26.729615 23790 tablet_bootstrap.cc:492] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.730654 23787 tablet_bootstrap.cc:492] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.731226 23787 ts_tablet_manager.cc:1403] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20251023 20:15:26.731878 23790 tablet_bootstrap.cc:654] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.732277 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 4c0f207603f241669afd4523b0f9f82a (DEFAULT_TABLE table=default.Table_64ad686282f94cb7ac869f59933669f1 [id=91b7c90942ea4aa8bb8e4b32e6d7c5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.732347 23787 raft_consensus.cc:359] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.732707 23787 raft_consensus.cc:385] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.732718 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0f207603f241669afd4523b0f9f82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.732857 23787 raft_consensus.cc:740] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.733224 23787 consensus_queue.cc:260] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.734056 23368 tablet_service.cc:1505] Processing CreateTablet for tablet b74d2e6d360c4200ae8740dddc206401 (DEFAULT_TABLE table=default.Table_2afbf83f519f4b7da2c4acc31ab18380 [id=6736a50ebad44c42985a5be788bfc43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.734263 23787 ts_tablet_manager.cc:1434] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:26.734433 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74d2e6d360c4200ae8740dddc206401. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.734933 23787 tablet_bootstrap.cc:492] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.736532 23787 tablet_bootstrap.cc:654] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.742116 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:26.745260 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe49ee5aaf784ef7a9078ff504ddee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.745836 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b4838644a9403693572350b1bdd0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.749189 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a4c8dcbdfcf4a15b8a4194ea04b93a0" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.749567 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0f207603f241669afd4523b0f9f82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.750042 23530 raft_consensus.cc:2468] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.751636 24363 ts_tablet_manager.cc:1929] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.751719 24363 log.cc:1199] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/7540354f8577482785e6341fa9022315
I20251023 20:15:26.752007 24363 ts_tablet_manager.cc:1950] T 7540354f8577482785e6341fa9022315 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.752975 24259 consensus_queue.cc:1048] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 20:15:26.753705 23530 raft_consensus.cc:3060] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.762308 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "925e53d9ea6f4c7baa38551eb917828d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:26.762514 23669 raft_consensus.cc:3060] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.763847 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 60) took 1154 ms. Trace:
I20251023 20:15:26.763942 24363 rpcz_store.cc:276] 1023 20:15:25.609283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.609568 (+ 285us) service_pool.cc:225] Handling call
1023 20:15:26.714353 (+1104785us) ts_tablet_manager.cc:1151] Deleting tablet 7540354f8577482785e6341fa9022315
1023 20:15:26.714362 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.746739 (+ 32377us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.751551 (+ 4812us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.763826 (+ 12275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104595}
I20251023 20:15:26.764652 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 7540354f8577482785e6341fa9022315 (table default.Table_410a6baf06a642b18af37bebddac662f [id=75026c75215e419983355275cf38f7f3]) successfully deleted
I20251023 20:15:26.764797 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "925e53d9ea6f4c7baa38551eb917828d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.765002 23669 raft_consensus.cc:2468] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.766134 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a708cd011802458aa0e2991fd4b4a2d3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:26.766302 23669 raft_consensus.cc:3060] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.768813 23669 raft_consensus.cc:2468] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.769065 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 61) took 1159 ms. Trace:
I20251023 20:15:26.769853 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 886838284a0548f6bb089f62c686165e (DEFAULT_TABLE table=default.Table_95fb2c2528f44d8d84f4cd67d9663502 [id=4867e660fb23438495400e364dba7b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.770085 24363 rpcz_store.cc:276] 1023 20:15:25.609923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.610088 (+ 165us) service_pool.cc:225] Handling call
1023 20:15:26.764180 (+1154092us) ts_tablet_manager.cc:1151] Deleting tablet 39a62c97e0a3459e91c24c1c2f92389a
1023 20:15:26.764189 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.769047 (+ 4858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153950}
I20251023 20:15:26.770303 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886838284a0548f6bb089f62c686165e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.771281 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a708cd011802458aa0e2991fd4b4a2d3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.771668 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 62) took 1161 ms. Trace:
W20251023 20:15:26.772292 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
I20251023 20:15:26.772490 24363 rpcz_store.cc:276] 1023 20:15:25.610399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.610669 (+ 270us) service_pool.cc:225] Handling call
1023 20:15:26.771554 (+1160885us) ts_tablet_manager.cc:1151] Deleting tablet a841ffdc6cdd432dac7afe2e2b53f224
1023 20:15:26.771564 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.771652 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160708}
W20251023 20:15:26.773269 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
I20251023 20:15:26.773614 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a7306245d7b41cfab795450761144f5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.773790 23665 raft_consensus.cc:2468] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:26.773768 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a7306245d7b41cfab795450761144f5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.774822 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 63) took 1163 ms. Trace:
I20251023 20:15:26.777122 24363 rpcz_store.cc:276] 1023 20:15:25.611009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.611161 (+ 152us) service_pool.cc:225] Handling call
1023 20:15:26.774692 (+1163531us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:26.774703 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.774805 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163355}
I20251023 20:15:26.777694 23790 tablet_bootstrap.cc:492] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.777846 23790 ts_tablet_manager.cc:1403] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
W20251023 20:15:26.778172 23333 leader_election.cc:343] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.778168 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
W20251023 20:15:26.778506 23333 leader_election.cc:343] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.778770 23333 leader_election.cc:343] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.778960 24363 tablet_replica.cc:333] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.779027 23333 leader_election.cc:304] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:26.779309 24363 raft_consensus.cc:2243] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.780313 23515 tablet_service.cc:1505] Processing CreateTablet for tablet c7a63d37788240f0a0c3dbfdd14edeac (DEFAULT_TABLE table=default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.780602 24363 raft_consensus.cc:2272] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.779516 23790 raft_consensus.cc:359] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.780738 23790 raft_consensus.cc:385] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.780864 23790 raft_consensus.cc:740] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.781533 24363 ts_tablet_manager.cc:1916] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.781284 23790 consensus_queue.cc:260] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.783353 23333 leader_election.cc:343] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.783507 23333 leader_election.cc:343] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.783640 23333 leader_election.cc:343] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.783926 24297 raft_consensus.cc:2804] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.784003 24297 raft_consensus.cc:493] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.784063 24297 raft_consensus.cc:3060] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.784426 23790 ts_tablet_manager.cc:1434] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251023 20:15:26.785621 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a63d37788240f0a0c3dbfdd14edeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.787180 24297 raft_consensus.cc:515] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.787699 24297 leader_election.cc:290] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.787844 24297 consensus_queue.cc:1048] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.794409 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8ba7ab28dfa4ac38e0b58ba2115495c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.794626 23396 raft_consensus.cc:2468] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:26.797920 23790 tablet_bootstrap.cc:492] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.800145 23790 tablet_bootstrap.cc:654] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.801718 23510 tablet_service.cc:1505] Processing CreateTablet for tablet fe49ee5aaf784ef7a9078ff504ddee70 (DEFAULT_TABLE table=default.Table_21f538f7569b437b98fe8a5274592880 [id=6a431dc140754c84af27e9bd3ce01a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.802184 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe49ee5aaf784ef7a9078ff504ddee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.805018 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 94b4838644a9403693572350b1bdd0e2 (DEFAULT_TABLE table=default.Table_222ab66f7150462aad2efe8d16484b94 [id=f52ec73cfcfe4b36a08422d69aabf321]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.805408 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94b4838644a9403693572350b1bdd0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.805756 23786 tablet_bootstrap.cc:492] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.805943 23786 ts_tablet_manager.cc:1403] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.001s
I20251023 20:15:26.806851 23786 raft_consensus.cc:359] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.807052 23786 raft_consensus.cc:385] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.807124 23786 raft_consensus.cc:740] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.809265 23786 consensus_queue.cc:260] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.810071 23790 tablet_bootstrap.cc:492] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.810214 23790 ts_tablet_manager.cc:1403] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20251023 20:15:26.810938 23790 raft_consensus.cc:359] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.811115 23790 raft_consensus.cc:385] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.811056 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 4c0f207603f241669afd4523b0f9f82a (DEFAULT_TABLE table=default.Table_64ad686282f94cb7ac869f59933669f1 [id=91b7c90942ea4aa8bb8e4b32e6d7c5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.811180 23790 raft_consensus.cc:740] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.811417 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0f207603f241669afd4523b0f9f82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.811385 23790 consensus_queue.cc:260] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.812927 24184 raft_consensus.cc:493] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.813109 24184 raft_consensus.cc:515] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.813797 24184 leader_election.cc:290] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.814304 23790 ts_tablet_manager.cc:1434] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:26.814589 23790 tablet_bootstrap.cc:492] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:26.815490 23530 raft_consensus.cc:3060] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.816252 23790 tablet_bootstrap.cc:654] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.816680 24349 ts_tablet_manager.cc:1929] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.816823 24349 log.cc:1199] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/8289ce37fa074891beb584bf2fff9747
W20251023 20:15:26.820832 23470 leader_election.cc:343] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:26.821432 23786 ts_tablet_manager.cc:1434] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251023 20:15:26.821554 23368 tablet_service.cc:1505] Processing CreateTablet for tablet fe49ee5aaf784ef7a9078ff504ddee70 (DEFAULT_TABLE table=default.Table_21f538f7569b437b98fe8a5274592880 [id=6a431dc140754c84af27e9bd3ce01a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.822016 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe49ee5aaf784ef7a9078ff504ddee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.822252 23786 tablet_bootstrap.cc:492] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.827518 23369 tablet_service.cc:1505] Processing CreateTablet for tablet c7a63d37788240f0a0c3dbfdd14edeac (DEFAULT_TABLE table=default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.827900 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a63d37788240f0a0c3dbfdd14edeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.782796 23646 tablet_service.cc:1505] Processing CreateTablet for tablet c7a63d37788240f0a0c3dbfdd14edeac (DEFAULT_TABLE table=default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:26.829385 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a63d37788240f0a0c3dbfdd14edeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:26.785471 24553 raft_consensus.cc:493] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.832532 24553 raft_consensus.cc:515] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.804172 24600 raft_consensus.cc:493] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.833242 24600 raft_consensus.cc:515] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.833825 24600 leader_election.cc:290] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.834261 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a4c8dcbdfcf4a15b8a4194ea04b93a0" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.834488 23532 raft_consensus.cc:3060] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.834666 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:26.835216 23605 leader_election.cc:343] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.835546 24553 leader_election.cc:290] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.836272 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.836635 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9c5b69df4c04365b1ac1457fe8300cc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.836701 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9c5b69df4c04365b1ac1457fe8300cc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.836858 23399 raft_consensus.cc:2468] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
W20251023 20:15:26.837137 23604 leader_election.cc:343] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.837239 23604 leader_election.cc:304] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.837255 23605 leader_election.cc:304] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:26.837603 23604 leader_election.cc:343] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.837561 23532 raft_consensus.cc:2468] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:26.837870 24553 raft_consensus.cc:2749] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.838032 24553 raft_consensus.cc:2804] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.838084 24553 raft_consensus.cc:493] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.838131 24553 raft_consensus.cc:3060] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.840517 24553 raft_consensus.cc:515] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.841550 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9c5b69df4c04365b1ac1457fe8300cc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.841718 23399 raft_consensus.cc:3060] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.842495 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9c5b69df4c04365b1ac1457fe8300cc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:26.844170 23399 raft_consensus.cc:2468] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
W20251023 20:15:26.844220 23604 leader_election.cc:343] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.844450 24553 leader_election.cc:290] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.844717 23605 leader_election.cc:304] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.845314 24553 raft_consensus.cc:2804] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.845450 24553 raft_consensus.cc:697] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:26.845724 24553 consensus_queue.cc:237] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.846572 24184 raft_consensus.cc:493] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.846741 24184 raft_consensus.cc:515] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.847334 24184 leader_election.cc:290] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.849166 23471 leader_election.cc:304] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:26.849646 24184 raft_consensus.cc:2804] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.849735 24184 raft_consensus.cc:493] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.849826 24184 raft_consensus.cc:3060] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.851629 24363 ts_tablet_manager.cc:1929] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:26.851730 24363 log.cc:1199] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/6e1b4748c0244b258960b1025da47562
I20251023 20:15:26.852021 24363 ts_tablet_manager.cc:1950] T 6e1b4748c0244b258960b1025da47562 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.852546 24184 raft_consensus.cc:515] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.852758 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 64) took 1241 ms. Trace:
I20251023 20:15:26.852847 24363 rpcz_store.cc:276] 1023 20:15:25.611620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.611815 (+ 195us) service_pool.cc:225] Handling call
1023 20:15:26.778912 (+1167097us) ts_tablet_manager.cc:1151] Deleting tablet 6e1b4748c0244b258960b1025da47562
1023 20:15:26.778922 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.832095 (+ 53173us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.851510 (+ 19415us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.852735 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1166859}
I20251023 20:15:26.853318 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 6e1b4748c0244b258960b1025da47562 (table default.Table_4740fc3fe4ac4927b7f776c980467298 [id=a93a658ff7104725a680f6ee3686e0a4]) successfully deleted
I20251023 20:15:26.853731 24184 leader_election.cc:290] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.857164 23787 tablet_bootstrap.cc:492] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.857293 23787 ts_tablet_manager.cc:1403] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.123s user 0.013s sys 0.017s
I20251023 20:15:26.857975 23787 raft_consensus.cc:359] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:26.858136 23787 raft_consensus.cc:385] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.858183 23787 raft_consensus.cc:740] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.858346 23787 consensus_queue.cc:260] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:26.861464 24658 raft_consensus.cc:493] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.862506 24658 raft_consensus.cc:515] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:26.863327 23470 leader_election.cc:343] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a8ba7ab28dfa4ac38e0b58ba2115495c
I20251023 20:15:26.863278 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:26.864601 23333 leader_election.cc:343] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3a4c8dcbdfcf4a15b8a4194ea04b93a0
I20251023 20:15:26.864909 23333 leader_election.cc:304] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.865288 24230 raft_consensus.cc:2804] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.865411 24230 raft_consensus.cc:697] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.865638 24230 consensus_queue.cc:237] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.868160 23787 ts_tablet_manager.cc:1434] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251023 20:15:26.868260 24621 raft_consensus.cc:493] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.868376 24621 raft_consensus.cc:515] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.868896 24621 leader_election.cc:290] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.869515 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:26.869729 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:26.869949 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:26.870157 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:26.870326 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:26.870537 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:26.870893 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:26.871058 24607 tablet.cc:1722] T 4f47ddde9f3d4042bc84853332e21950 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:26.871518 24619 tablet.cc:1722] T 4f47ddde9f3d4042bc84853332e21950 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:26.875212 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:26.875711 23786 tablet_bootstrap.cc:654] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.877763 23787 tablet_bootstrap.cc:492] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.879894 23787 tablet_bootstrap.cc:654] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.879920 24621 raft_consensus.cc:493] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.880053 24621 raft_consensus.cc:515] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.880606 24621 leader_election.cc:290] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:26.880595 24184 raft_consensus.cc:493] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.880713 24184 raft_consensus.cc:515] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.881975 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a64a7c2e1f242e1b54243489622bae5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:26.882508 23605 leader_election.cc:343] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.883575 24658 leader_election.cc:290] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.886503 23333 leader_election.cc:304] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.886835 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:26.888003 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:26.888279 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:26.888795 24555 tablet.cc:1722] T 4f47ddde9f3d4042bc84853332e21950 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:26.888854 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:26.888996 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:26.889045 24495 raft_consensus.cc:2804] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:26.889179 24495 raft_consensus.cc:697] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:26.889405 24495 consensus_queue.cc:237] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.889767 24184 leader_election.cc:290] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.890594 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:26.890760 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:26.890921 23668 raft_consensus.cc:2393] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:26.890967 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:26.891134 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:26.891510 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:26.891697 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:26.891886 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:26.892023 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:26.892333 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:26.892709 23333 leader_election.cc:304] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.892665 23212 catalog_manager.cc:5649] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:26.893289 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:26.893636 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]). Expected version 1 got 0
I20251023 20:15:26.894196 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:26.894544 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:26.894687 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:26.894879 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:26.895082 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
I20251023 20:15:26.895354 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:26.895568 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:26.895990 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:26.896486 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:26.896736 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:26.896881 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:26.897289 24363 tablet_replica.cc:333] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.898109 23786 tablet_bootstrap.cc:492] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.898248 23786 ts_tablet_manager.cc:1403] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.002s
I20251023 20:15:26.899014 23786 raft_consensus.cc:359] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.899195 23786 raft_consensus.cc:385] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.899257 23786 raft_consensus.cc:740] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.899438 23786 consensus_queue.cc:260] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.900141 23786 ts_tablet_manager.cc:1434] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:26.900388 23786 tablet_bootstrap.cc:492] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.901238 24363 raft_consensus.cc:2243] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:26.901571 24363 raft_consensus.cc:2272] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.902133 24363 ts_tablet_manager.cc:1916] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.902215 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a64a7c2e1f242e1b54243489622bae5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:26.902712 23604 leader_election.cc:343] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.902840 23604 leader_election.cc:304] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.903554 24658 raft_consensus.cc:2749] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:26.909356 24297 consensus_queue.cc:1048] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.910211 24573 raft_consensus.cc:2804] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.910300 24573 raft_consensus.cc:493] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.910369 24573 raft_consensus.cc:3060] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.911185 23786 tablet_bootstrap.cc:654] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.921882 24658 raft_consensus.cc:493] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.922142 24658 raft_consensus.cc:515] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.923278 24658 leader_election.cc:290] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:26.923260 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2a27706ecdb4ab78bb96c10a9f6757f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:26.923817 24184 raft_consensus.cc:493] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.923949 24184 raft_consensus.cc:515] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.924492 24184 leader_election.cc:290] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:26.924827 23604 leader_election.cc:343] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:26.925946 23470 leader_election.cc:343] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:26.931679 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet dcbacd8e23bc4b129757d9faf89ce5b4 (table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]). Expected version 1 got 0
I20251023 20:15:26.932511 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:26.933276 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:26.935212 23786 tablet_bootstrap.cc:492] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.937714 23532 raft_consensus.cc:3060] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.944039 23787 tablet_bootstrap.cc:492] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.944185 23787 ts_tablet_manager.cc:1403] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.010s
I20251023 20:15:26.944948 23787 raft_consensus.cc:359] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:26.945148 23787 raft_consensus.cc:385] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.945202 23787 raft_consensus.cc:740] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.945384 23787 consensus_queue.cc:260] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:26.947642 24553 consensus_queue.cc:1048] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:26.948915 23786 ts_tablet_manager.cc:1403] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.004s
I20251023 20:15:26.950605 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]). Expected version 2 got 0
I20251023 20:15:26.951867 23786 raft_consensus.cc:359] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.952481 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]). Expected version 2 got 0
I20251023 20:15:26.953732 23787 ts_tablet_manager.cc:1434] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:26.954041 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:26.954223 24573 raft_consensus.cc:515] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.954489 23786 raft_consensus.cc:385] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.954558 23786 raft_consensus.cc:740] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.954864 24573 leader_election.cc:290] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.954807 23786 consensus_queue.cc:260] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.955314 23786 ts_tablet_manager.cc:1434] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251023 20:15:26.955602 23786 tablet_bootstrap.cc:492] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:26.956013 23787 tablet_bootstrap.cc:492] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.957666 23787 tablet_bootstrap.cc:654] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.958706 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]). Expected version 3 got 0
I20251023 20:15:26.960032 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]). Expected version 4 got 0
I20251023 20:15:26.961627 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]). Expected version 2 got 0
I20251023 20:15:26.963020 23212 catalog_manager.cc:5649] T 147781e046fe4feb9815147c774fe210 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:26.965126 23786 tablet_bootstrap.cc:654] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.967725 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:26.967831 24363 ts_tablet_manager.cc:1929] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:26.968036 24363 log.cc:1199] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a2dbb7997d0f4e6bac48cfb196f5ccb6
I20251023 20:15:26.968442 24363 ts_tablet_manager.cc:1950] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:26.969321 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 65) took 1357 ms. Trace:
I20251023 20:15:26.969590 24363 rpcz_store.cc:276] 1023 20:15:25.612162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.612316 (+ 154us) service_pool.cc:225] Handling call
1023 20:15:26.897232 (+1284916us) ts_tablet_manager.cc:1151] Deleting tablet a2dbb7997d0f4e6bac48cfb196f5ccb6
1023 20:15:26.897243 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.931973 (+ 34730us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.967729 (+ 35756us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.969298 (+ 1569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1284769}
I20251023 20:15:26.969826 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 377de4ea0bd6489284fac6b0fab52c8c (table default.Table_ede581d1b2824c17933809e745e8ab07 [id=ee282decb79a4f27908c253cf524f971]). Expected version 1 got 0
I20251023 20:15:26.970243 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:26.970427 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:26.970631 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]). Expected version 3 got 0
W20251023 20:15:26.970654 23333 leader_election.cc:343] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3a4c8dcbdfcf4a15b8a4194ea04b93a0
I20251023 20:15:26.970993 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:26.971390 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
W20251023 20:15:26.971493 23333 consensus_peers.cc:597] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d1ddde7e3134e23b3a9356523a93753. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:26.976188 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed46b7e67e4e4ae7b30ea69f1d3fdc2f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.977208 23786 tablet_bootstrap.cc:492] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:26.977247 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 (table default.Table_e186a061f84042cd82645b37beae1da1 [id=dc553d161a13467c99df49720181dba1]) successfully deleted
I20251023 20:15:26.977344 23786 ts_tablet_manager.cc:1403] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251023 20:15:26.977802 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 66) took 1360 ms. Trace:
I20251023 20:15:26.978433 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147781e046fe4feb9815147c774fe210" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.978611 24363 rpcz_store.cc:276] 1023 20:15:25.617462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.617503 (+ 41us) service_pool.cc:225] Handling call
1023 20:15:26.977660 (+1360157us) ts_tablet_manager.cc:1151] Deleting tablet 453eea5cdd2e47f29577c61b40ee59c0
1023 20:15:26.977672 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.977780 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1356481}
W20251023 20:15:26.979144 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
I20251023 20:15:26.979409 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2a27706ecdb4ab78bb96c10a9f6757f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.979554 23399 raft_consensus.cc:2468] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:26.979918 24363 tablet_replica.cc:333] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:26.980010 23790 tablet_bootstrap.cc:492] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:26.980127 23790 ts_tablet_manager.cc:1403] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.166s user 0.000s sys 0.021s
I20251023 20:15:26.980341 24363 raft_consensus.cc:2243] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:26.980568 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8ba7ab28dfa4ac38e0b58ba2115495c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.980713 23396 raft_consensus.cc:3060] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.980866 24363 raft_consensus.cc:2272] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:26.982833 24363 ts_tablet_manager.cc:1916] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:26.983219 23396 raft_consensus.cc:2468] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:26.983490 24670 raft_consensus.cc:493] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:26.983618 24670 raft_consensus.cc:515] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.984153 24670 leader_election.cc:290] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:26.984632 23605 leader_election.cc:304] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:26.983728 23787 tablet_bootstrap.cc:492] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:26.984922 23787 ts_tablet_manager.cc:1403] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20251023 20:15:26.985015 24659 raft_consensus.cc:2804] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:26.985114 24659 raft_consensus.cc:493] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:26.985183 24659 raft_consensus.cc:3060] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.985952 23787 raft_consensus.cc:359] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:26.986155 23787 raft_consensus.cc:385] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.986235 23787 raft_consensus.cc:740] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.986482 23787 consensus_queue.cc:260] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:26.987623 24659 raft_consensus.cc:515] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.988137 24659 leader_election.cc:290] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:26.989439 23203 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.189:33509, user_credentials={real_user=slave}} blocked reactor thread for 57405.5us
I20251023 20:15:26.989765 23787 ts_tablet_manager.cc:1434] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:26.990029 23787 tablet_bootstrap.cc:492] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:26.990078 23786 raft_consensus.cc:359] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.990234 23786 raft_consensus.cc:385] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:26.990280 23786 raft_consensus.cc:740] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:26.990456 23786 consensus_queue.cc:260] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:26.990893 23786 ts_tablet_manager.cc:1434] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.013s user 0.000s sys 0.000s
I20251023 20:15:26.991150 23786 tablet_bootstrap.cc:492] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:26.991526 23206 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2a27706ecdb4ab78bb96c10a9f6757f consensus state has no leader
I20251023 20:15:26.991736 23787 tablet_bootstrap.cc:654] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:26.992939 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2a27706ecdb4ab78bb96c10a9f6757f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:26.993515 23604 leader_election.cc:343] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:26.996528 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2a27706ecdb4ab78bb96c10a9f6757f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:26.996583 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fa6a838b1bd409ba2a4e2074576ee23" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.996701 23397 raft_consensus.cc:3060] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:26.997551 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cec51c54ea534b1da4b1d170e79d9f1c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:26.999256 23397 raft_consensus.cc:2468] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:27.004457 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15ced74d0dd74e5195ed98358957d0dd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.004645 23668 raft_consensus.cc:2468] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.012585 24659 consensus_queue.cc:1048] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.013628 23605 leader_election.cc:304] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.014254 24659 raft_consensus.cc:2804] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.014418 24659 raft_consensus.cc:697] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:27.014736 24659 consensus_queue.cc:237] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.018736 23790 raft_consensus.cc:359] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:26.996672 24349 ts_tablet_manager.cc:1950] T 8289ce37fa074891beb584bf2fff9747 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:26.998063 23786 tablet_bootstrap.cc:654] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.005924 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15ced74d0dd74e5195ed98358957d0dd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.018677 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a7306245d7b41cfab795450761144f5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:27.019605 23471 leader_election.cc:304] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.020927 24259 raft_consensus.cc:2749] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.021183 23471 leader_election.cc:304] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.021212 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a7306245d7b41cfab795450761144f5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.021374 23665 raft_consensus.cc:3060] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.021548 23471 leader_election.cc:304] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:27.021893 24615 raft_consensus.cc:2749] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.022039 24615 raft_consensus.cc:2804] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.022117 24615 raft_consensus.cc:697] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:27.022336 24615 consensus_queue.cc:237] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.023921 23665 raft_consensus.cc:2468] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.001623 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d82c92e863b2470699cc822b8cb8c14d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.027760 24607 tablet.cc:1722] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:27.028182 24259 raft_consensus.cc:493] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.028326 24259 raft_consensus.cc:515] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.028496 24619 tablet.cc:1722] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:27.028873 24259 leader_election.cc:290] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:27.031515 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 74) took 1046 ms. Trace:
I20251023 20:15:27.034267 23790 raft_consensus.cc:385] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.034361 23790 raft_consensus.cc:740] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.034559 23790 consensus_queue.cc:260] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.035096 23790 ts_tablet_manager.cc:1434] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.055s user 0.000s sys 0.001s
I20251023 20:15:27.035378 23790 tablet_bootstrap.cc:492] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:27.031673 24349 rpcz_store.cc:276] 1023 20:15:25.985457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.985494 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:26.554010 (+568516us) ts_tablet_manager.cc:1151] Deleting tablet 8289ce37fa074891beb584bf2fff9747
1023 20:15:26.554015 (+ 5us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:26.774387 (+220372us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:26.816550 (+ 42163us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.031496 (+214946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":568371}
I20251023 20:15:27.038968 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 75) took 1053 ms. Trace:
I20251023 20:15:27.039048 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 8289ce37fa074891beb584bf2fff9747 (table default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]) successfully deleted
I20251023 20:15:27.041633 24349 rpcz_store.cc:276] 1023 20:15:25.985849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.985887 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:27.038837 (+1052950us) ts_tablet_manager.cc:1151] Deleting tablet 40c659c273c449939d0dd708d39b393f
1023 20:15:27.038848 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.038950 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052781}
W20251023 20:15:27.042879 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
I20251023 20:15:27.043567 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 76) took 1056 ms. Trace:
I20251023 20:15:27.043665 24349 rpcz_store.cc:276] 1023 20:15:25.986207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.986245 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:27.042936 (+1056691us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:27.042947 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.043042 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1056532}
W20251023 20:15:27.044062 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
I20251023 20:15:27.044696 24349 tablet_replica.cc:333] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.044898 24349 raft_consensus.cc:2243] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.045055 24349 raft_consensus.cc:2272] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.045287 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.045564 23665 raft_consensus.cc:2393] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:27.046929 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.047636 23471 leader_election.cc:304] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.048116 24259 raft_consensus.cc:3060] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.049413 24349 ts_tablet_manager.cc:1916] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.049525 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.049708 23665 raft_consensus.cc:2468] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.051220 24259 raft_consensus.cc:2749] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.051822 24259 consensus_queue.cc:1048] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.051934 23471 leader_election.cc:304] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.052192 24184 raft_consensus.cc:2749] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.059324 23333 leader_election.cc:304] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:27.059593 24670 raft_consensus.cc:2804] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.059720 24670 raft_consensus.cc:697] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.062299 24600 consensus_queue.cc:1048] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.062626 24670 consensus_queue.cc:237] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.068562 24363 ts_tablet_manager.cc:1929] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.068744 24363 log.cc:1199] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/edec01a63a3046d9ae5a372fc9068b52
I20251023 20:15:27.069200 24363 ts_tablet_manager.cc:1950] T edec01a63a3046d9ae5a372fc9068b52 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:27.070631 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 69) took 1448 ms. Trace:
I20251023 20:15:27.072516 23786 tablet_bootstrap.cc:492] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.072736 23786 ts_tablet_manager.cc:1403] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.082s user 0.016s sys 0.000s
W20251023 20:15:27.074322 23333 leader_election.cc:343] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:27.074826 23333 leader_election.cc:343] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.073971 23786 raft_consensus.cc:359] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:27.075064 23333 leader_election.cc:343] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:27.075189 23333 leader_election.cc:343] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.075198 23786 raft_consensus.cc:385] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.075342 23786 raft_consensus.cc:740] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.075609 23786 consensus_queue.cc:260] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.076246 23786 ts_tablet_manager.cc:1434] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:27.077600 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe922c438a894a7aa04699b2d2897a28" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.078925 23786 tablet_bootstrap.cc:492] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.080680 24670 raft_consensus.cc:493] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.080824 24670 raft_consensus.cc:515] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.081416 24670 leader_election.cc:290] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:27.081950 24555 tablet.cc:1722] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:27.082540 23790 tablet_bootstrap.cc:654] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.083081 24363 rpcz_store.cc:276] 1023 20:15:25.621704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.621743 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:26.979873 (+1358130us) ts_tablet_manager.cc:1151] Deleting tablet edec01a63a3046d9ae5a372fc9068b52
1023 20:15:26.979883 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.008549 (+ 28666us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.068430 (+ 59881us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.070608 (+ 2178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357925}
I20251023 20:15:27.093257 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet edec01a63a3046d9ae5a372fc9068b52 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
W20251023 20:15:27.093662 23333 leader_election.cc:343] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:27.093791 23333 leader_election.cc:304] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.094090 24670 raft_consensus.cc:2749] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.094512 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 68) took 1473 ms. Trace:
I20251023 20:15:27.094195 23333 leader_election.cc:304] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.095942 24670 raft_consensus.cc:2804] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.099733 24670 raft_consensus.cc:493] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.099869 24670 raft_consensus.cc:3060] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.100142 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:27.101099 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
W20251023 20:15:27.097230 23604 consensus_peers.cc:597] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.101383 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:27.101516 24363 rpcz_store.cc:276] 1023 20:15:25.621603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.622034 (+ 431us) service_pool.cc:225] Handling call
1023 20:15:27.094376 (+1472342us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:27.094387 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.094490 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472149}
I20251023 20:15:27.101754 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:27.102733 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 67) took 1481 ms. Trace:
I20251023 20:15:27.103384 23787 tablet_bootstrap.cc:492] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.103864 23786 tablet_bootstrap.cc:654] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.104553 24349 ts_tablet_manager.cc:1929] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.104661 24349 log.cc:1199] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/c95db7b5b9fd46acae8ffa7119915907
W20251023 20:15:27.104840 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
I20251023 20:15:27.105127 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:27.105381 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:27.105350 24659 consensus_queue.cc:1048] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.103899 23787 ts_tablet_manager.cc:1403] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.114s user 0.005s sys 0.005s
I20251023 20:15:27.107193 23787 raft_consensus.cc:359] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.107528 23787 raft_consensus.cc:385] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.107656 23787 raft_consensus.cc:740] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.107873 23790 tablet_bootstrap.cc:492] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:27.107898 23787 consensus_queue.cc:260] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.108016 23790 ts_tablet_manager.cc:1403] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.002s
I20251023 20:15:27.108729 23790 raft_consensus.cc:359] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.108906 23790 raft_consensus.cc:385] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.108950 23790 raft_consensus.cc:740] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.109158 23790 consensus_queue.cc:260] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.110169 23790 ts_tablet_manager.cc:1434] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:27.110582 23790 tablet_bootstrap.cc:492] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:27.112164 23787 ts_tablet_manager.cc:1434] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:27.112787 23790 tablet_bootstrap.cc:654] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.113042 23787 tablet_bootstrap.cc:492] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.114606 24363 rpcz_store.cc:276] 1023 20:15:25.621311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.622246 (+ 935us) service_pool.cc:225] Handling call
1023 20:15:27.102607 (+1480361us) ts_tablet_manager.cc:1151] Deleting tablet d9f73477a06542c0b215ae86716cdc4e
1023 20:15:27.102618 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.102715 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479819}
I20251023 20:15:27.115054 24363 tablet_replica.cc:333] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:27.115135 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
I20251023 20:15:27.115227 24363 raft_consensus.cc:2243] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.115365 24363 raft_consensus.cc:2272] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.117746 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:27.118307 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:27.119056 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:27.119256 24659 consensus_queue.cc:1048] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.119671 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:27.119839 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:27.120132 23787 tablet_bootstrap.cc:654] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.121428 24363 ts_tablet_manager.cc:1916] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.121743 24670 raft_consensus.cc:515] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.122323 24670 leader_election.cc:290] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.096287 23333 leader_election.cc:304] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.122787 23333 leader_election.cc:304] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.123165 24621 raft_consensus.cc:2804] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.123286 24621 raft_consensus.cc:697] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.123350 24297 raft_consensus.cc:2804] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.123437 24297 raft_consensus.cc:493] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.123489 24621 consensus_queue.cc:237] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.123498 24297 raft_consensus.cc:3060] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.126358 24297 raft_consensus.cc:515] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.126909 24297 leader_election.cc:290] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:27.128767 24349 ts_tablet_manager.cc:1950] T c95db7b5b9fd46acae8ffa7119915907 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.131042 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 77) took 1144 ms. Trace:
I20251023 20:15:27.132395 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:27.131798 23532 raft_consensus.cc:3060] T 147781e046fe4feb9815147c774fe210 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.132718 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:27.132975 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:27.133188 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:27.133688 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:27.132798 24363 ts_tablet_manager.cc:1929] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.133844 24363 log.cc:1199] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/0858358aabff47a69dc77f4669b225c5
I20251023 20:15:27.133874 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:27.134132 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:27.134331 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:27.134439 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:27.134680 24363 ts_tablet_manager.cc:1950] T 0858358aabff47a69dc77f4669b225c5 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:27.134871 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:27.135017 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:27.135174 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:27.135412 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:27.135360 23471 leader_election.cc:304] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.135416 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 72) took 1499 ms. Trace:
I20251023 20:15:27.135486 24363 rpcz_store.cc:276] 1023 20:15:25.635720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.635761 (+ 41us) service_pool.cc:225] Handling call
1023 20:15:27.115009 (+1479248us) ts_tablet_manager.cc:1151] Deleting tablet 0858358aabff47a69dc77f4669b225c5
1023 20:15:27.115019 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.124870 (+ 9851us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.132690 (+ 7820us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.135396 (+ 2706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470046}
I20251023 20:15:27.135645 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:27.135753 24615 raft_consensus.cc:3060] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.135964 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 0858358aabff47a69dc77f4669b225c5 (table default.Table_c9ccf1142a9b449f91e99360376a7b22 [id=c36b140eefe1435d93b7ba48ca07e4a1]) successfully deleted
I20251023 20:15:27.135994 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:27.136159 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:27.136843 23787 tablet_bootstrap.cc:492] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.137027 23787 ts_tablet_manager.cc:1403] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.004s
I20251023 20:15:27.137959 23787 raft_consensus.cc:359] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.138231 23787 raft_consensus.cc:385] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.138326 23787 raft_consensus.cc:740] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.138581 23787 consensus_queue.cc:260] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.138911 24615 raft_consensus.cc:2749] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.139227 23787 ts_tablet_manager.cc:1434] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:27.139885 24349 rpcz_store.cc:276] 1023 20:15:25.986947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.987007 (+ 60us) service_pool.cc:225] Handling call
1023 20:15:27.044647 (+1057640us) ts_tablet_manager.cc:1151] Deleting tablet c95db7b5b9fd46acae8ffa7119915907
1023 20:15:27.044657 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.052243 (+ 7586us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.104424 (+ 52181us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.131021 (+ 26597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057436}
I20251023 20:15:27.141521 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:27.142546 23787 tablet_bootstrap.cc:492] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.142658 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:27.143648 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c8403b53f04a8e97ffe058e59b425c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.143818 23668 raft_consensus.cc:2468] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.145026 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15ced74d0dd74e5195ed98358957d0dd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.145181 23665 raft_consensus.cc:3060] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:27.145961 23333 consensus_peers.cc:597] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7301d6121c9d4c8dbc642e895aa4e9f6. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.142949 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c8403b53f04a8e97ffe058e59b425c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.144968 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15ced74d0dd74e5195ed98358957d0dd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:27.147025 24349 tablet_replica.cc:333] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.147186 24349 raft_consensus.cc:2243] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:27.147323 23333 consensus_peers.cc:597] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fc7e18054c440098dc2470143ea7fae. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.147624 23665 raft_consensus.cc:2468] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.147792 24621 consensus_queue.cc:1048] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.147949 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:27.148245 24349 raft_consensus.cc:2272] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.148711 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 71) took 1513 ms. Trace:
I20251023 20:15:27.149015 24349 ts_tablet_manager.cc:1916] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.148990 24363 rpcz_store.cc:276] 1023 20:15:25.635545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.636119 (+ 574us) service_pool.cc:225] Handling call
1023 20:15:27.139450 (+1503331us) ts_tablet_manager.cc:1151] Deleting tablet 370a790a053446b2b06314c7db242270
1023 20:15:27.148583 (+ 9133us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.148691 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494435}
W20251023 20:15:27.149108 23333 leader_election.cc:343] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:27.149174 23604 consensus_peers.cc:597] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.148938 23787 tablet_bootstrap.cc:654] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:27.149298 23333 leader_election.cc:343] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.149644 23333 leader_election.cc:304] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.150612 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.150774 23668 raft_consensus.cc:3060] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.152918 24363 tablet_replica.cc:333] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:27.153115 24363 raft_consensus.cc:2243] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.153260 24363 raft_consensus.cc:2272] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.153931 24621 raft_consensus.cc:2804] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.154013 24621 raft_consensus.cc:493] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.154074 24621 raft_consensus.cc:3060] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.154161 24363 ts_tablet_manager.cc:1916] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.156883 24621 raft_consensus.cc:515] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.157788 24621 leader_election.cc:290] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:27.158947 23668 raft_consensus.cc:2468] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.160287 24621 consensus_queue.cc:1048] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.160889 23333 leader_election.cc:304] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.160954 24184 consensus_queue.cc:1048] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.161394 24621 raft_consensus.cc:2804] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.161501 24621 raft_consensus.cc:697] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.161743 24621 consensus_queue.cc:237] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.163213 23787 tablet_bootstrap.cc:492] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.163354 23787 ts_tablet_manager.cc:1403] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20251023 20:15:27.164319 23787 raft_consensus.cc:359] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.164474 23787 raft_consensus.cc:385] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.164654 23787 raft_consensus.cc:740] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:27.166759 23604 consensus_peers.cc:597] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.164863 23787 consensus_queue.cc:260] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.168462 23333 leader_election.cc:304] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:27.168824 24670 raft_consensus.cc:2804] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.168432 23787 ts_tablet_manager.cc:1434] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:27.168946 24670 raft_consensus.cc:697] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.169447 24670 consensus_queue.cc:237] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.170591 23787 tablet_bootstrap.cc:492] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.146744 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet c95db7b5b9fd46acae8ffa7119915907 (table default.Table_91596859556249fbb6e82b7dc5756a92 [id=5dff7019113a4323ae78b11f6b547b53]) successfully deleted
W20251023 20:15:27.174196 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
I20251023 20:15:27.174585 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
I20251023 20:15:27.174964 23786 tablet_bootstrap.cc:492] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.175093 23786 ts_tablet_manager.cc:1403] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.099s user 0.005s sys 0.002s
I20251023 20:15:27.176340 23790 tablet_bootstrap.cc:492] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:27.176504 23790 ts_tablet_manager.cc:1403] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.003s
I20251023 20:15:27.177021 23787 tablet_bootstrap.cc:654] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.177304 23790 raft_consensus.cc:359] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.177501 23790 raft_consensus.cc:385] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.177665 23790 raft_consensus.cc:740] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.178341 23786 raft_consensus.cc:359] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.178519 23786 raft_consensus.cc:385] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.178581 23786 raft_consensus.cc:740] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.179200 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c8403b53f04a8e97ffe058e59b425c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:27.180084 23333 leader_election.cc:343] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.180325 23786 consensus_queue.cc:260] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.180886 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37c8403b53f04a8e97ffe058e59b425c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.181037 23665 raft_consensus.cc:3060] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.181324 24363 ts_tablet_manager.cc:1929] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.181401 24363 log.cc:1199] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/5964da70989b4e6cb77fe5d0a52ce9cf
I20251023 20:15:27.181674 24363 ts_tablet_manager.cc:1950] T 5964da70989b4e6cb77fe5d0a52ce9cf P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:27.184999 23665 raft_consensus.cc:2468] T 37c8403b53f04a8e97ffe058e59b425c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.185352 23786 ts_tablet_manager.cc:1434] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251023 20:15:27.187472 23786 tablet_bootstrap.cc:492] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:27.188277 23333 leader_election.cc:343] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.190276 23787 tablet_bootstrap.cc:492] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.190389 23787 ts_tablet_manager.cc:1403] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20251023 20:15:27.191066 23787 raft_consensus.cc:359] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.191222 23787 raft_consensus.cc:385] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.191285 23787 raft_consensus.cc:740] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.193218 23787 consensus_queue.cc:260] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.193476 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 73) took 1557 ms. Trace:
I20251023 20:15:27.193622 24349 ts_tablet_manager.cc:1929] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.193769 24349 log.cc:1199] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/edec01a63a3046d9ae5a372fc9068b52
I20251023 20:15:27.193868 24363 rpcz_store.cc:276] 1023 20:15:25.636080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.636702 (+ 622us) service_pool.cc:225] Handling call
1023 20:15:27.152875 (+1516173us) ts_tablet_manager.cc:1151] Deleting tablet 5964da70989b4e6cb77fe5d0a52ce9cf
1023 20:15:27.152883 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.160465 (+ 7582us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.181227 (+ 20762us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.193456 (+ 12229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507813}
I20251023 20:15:27.194423 24349 ts_tablet_manager.cc:1950] T edec01a63a3046d9ae5a372fc9068b52 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.194404 23787 ts_tablet_manager.cc:1434] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:27.194797 23787 tablet_bootstrap.cc:492] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.196058 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 74) took 1559 ms. Trace:
I20251023 20:15:27.196396 24363 rpcz_store.cc:276] 1023 20:15:25.636962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.637265 (+ 303us) service_pool.cc:225] Handling call
1023 20:15:27.195918 (+1558653us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:27.195928 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.196038 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550796}
I20251023 20:15:27.197142 23787 tablet_bootstrap.cc:654] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.197355 24363 tablet_replica.cc:333] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:27.197592 24363 raft_consensus.cc:2243] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.197793 24363 raft_consensus.cc:2272] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.198035 23790 consensus_queue.cc:260] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.198489 23790 ts_tablet_manager.cc:1434] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251023 20:15:27.200011 23786 tablet_bootstrap.cc:654] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.200630 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 78) took 1213 ms. Trace:
I20251023 20:15:27.200752 24349 rpcz_store.cc:276] 1023 20:15:25.987347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.987451 (+ 104us) service_pool.cc:225] Handling call
1023 20:15:27.146979 (+1159528us) ts_tablet_manager.cc:1151] Deleting tablet edec01a63a3046d9ae5a372fc9068b52
1023 20:15:27.146988 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.176840 (+ 29852us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.192838 (+ 15998us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.200610 (+ 7772us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159334}
I20251023 20:15:27.200888 23333 leader_election.cc:304] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.201375 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 79) took 1213 ms. Trace:
I20251023 20:15:27.201486 24349 rpcz_store.cc:276] 1023 20:15:25.988280 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.988324 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:27.201264 (+1212940us) ts_tablet_manager.cc:1151] Deleting tablet 453eea5cdd2e47f29577c61b40ee59c0
1023 20:15:27.201272 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.201359 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1212691}
I20251023 20:15:27.201783 24363 ts_tablet_manager.cc:1916] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.202642 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 80) took 1213 ms. Trace:
I20251023 20:15:27.202754 24349 rpcz_store.cc:276] 1023 20:15:25.989139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.989180 (+ 41us) service_pool.cc:225] Handling call
1023 20:15:27.202567 (+1213387us) ts_tablet_manager.cc:1151] Deleting tablet f4778436aed64b53adb2bab54bcf263b
1023 20:15:27.202575 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.202627 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213098}
I20251023 20:15:27.203146 24349 tablet_replica.cc:333] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.203341 24349 raft_consensus.cc:2243] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:27.203651 24349 raft_consensus.cc:2272] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.204370 24349 ts_tablet_manager.cc:1916] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.204900 23787 tablet_bootstrap.cc:492] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.205032 23787 ts_tablet_manager.cc:1403] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20251023 20:15:27.205689 24621 raft_consensus.cc:2804] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.205857 24621 raft_consensus.cc:697] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.205749 23787 raft_consensus.cc:359] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.205992 23787 raft_consensus.cc:385] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.206041 23787 raft_consensus.cc:740] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.206082 24621 consensus_queue.cc:237] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.206244 23787 consensus_queue.cc:260] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.207155 23790 tablet_bootstrap.cc:492] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:27.207641 23787 ts_tablet_manager.cc:1434] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:27.208526 23787 tablet_bootstrap.cc:492] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.210670 23790 tablet_bootstrap.cc:654] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.211841 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 5964da70989b4e6cb77fe5d0a52ce9cf (table default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]) successfully deleted
I20251023 20:15:27.212239 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet edec01a63a3046d9ae5a372fc9068b52 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
W20251023 20:15:27.212960 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
W20251023 20:15:27.213294 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f4778436aed64b53adb2bab54bcf263b because the tablet was not found. No further retry: Not found: Tablet not found: f4778436aed64b53adb2bab54bcf263b
W20251023 20:15:27.213510 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
I20251023 20:15:27.213758 23787 tablet_bootstrap.cc:654] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.216126 24349 ts_tablet_manager.cc:1929] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.217500 23787 tablet_bootstrap.cc:492] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.217633 23787 ts_tablet_manager.cc:1403] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20251023 20:15:27.218309 23787 raft_consensus.cc:359] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.218528 23787 raft_consensus.cc:385] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.216467 23532 raft_consensus.cc:3060] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.218590 23787 raft_consensus.cc:740] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.218806 23787 consensus_queue.cc:260] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.220088 23787 ts_tablet_manager.cc:1434] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:27.220927 23787 tablet_bootstrap.cc:492] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.221777 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:27.222422 23787 tablet_bootstrap.cc:654] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.222549 24615 raft_consensus.cc:493] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.222674 24615 raft_consensus.cc:515] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.223001 24349 log.cc:1199] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:27.223255 24615 leader_election.cc:290] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:27.223357 24349 ts_tablet_manager.cc:1950] T 79e3d422a66f432789c0cf4a00e57af2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.223786 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.224027 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 81) took 1234 ms. Trace:
I20251023 20:15:27.224720 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86b9f4c5d9e0469c87071a33722ce3eb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.224979 23397 raft_consensus.cc:2393] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:27.225400 23471 leader_election.cc:304] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.233384 23786 tablet_bootstrap.cc:492] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.233534 24615 raft_consensus.cc:3060] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.233660 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:27.233632 24349 rpcz_store.cc:276] 1023 20:15:25.989635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.989671 (+ 36us) service_pool.cc:225] Handling call
1023 20:15:27.203108 (+1213437us) ts_tablet_manager.cc:1151] Deleting tablet 79e3d422a66f432789c0cf4a00e57af2
1023 20:15:27.203116 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.211092 (+ 7976us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.216032 (+ 4940us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.224010 (+ 7978us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213238}
I20251023 20:15:27.234072 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:27.234138 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 79e3d422a66f432789c0cf4a00e57af2 (table default.Table_7b6d17b4f21740f0a5dadc35a23c919f [id=8b9eb030f1a24f68bb8e2e87b580727a]) successfully deleted
I20251023 20:15:27.234512 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 82) took 1244 ms. Trace:
I20251023 20:15:27.234584 24349 rpcz_store.cc:276] 1023 20:15:25.989985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.990018 (+ 33us) service_pool.cc:225] Handling call
1023 20:15:27.234367 (+1244349us) ts_tablet_manager.cc:1151] Deleting tablet d951134f6773410386bc57f7f90310c3
1023 20:15:27.234378 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.234495 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244161}
I20251023 20:15:27.234684 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
W20251023 20:15:27.235002 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d951134f6773410386bc57f7f90310c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d951134f6773410386bc57f7f90310c3 already in progress: creating tablet
I20251023 20:15:27.235302 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:27.235333 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 83) took 1245 ms. Trace:
I20251023 20:15:27.235409 24349 rpcz_store.cc:276] 1023 20:15:25.990304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.990337 (+ 33us) service_pool.cc:225] Handling call
1023 20:15:27.235220 (+1244883us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:27.235229 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.235317 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244668}
I20251023 20:15:27.235584 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
W20251023 20:15:27.235785 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
I20251023 20:15:27.236023 24349 tablet_replica.cc:333] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.236186 24615 raft_consensus.cc:2749] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.233529 23786 ts_tablet_manager.cc:1403] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
I20251023 20:15:27.237644 23786 raft_consensus.cc:359] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.237857 23786 raft_consensus.cc:385] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.237931 23786 raft_consensus.cc:740] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.238276 23786 consensus_queue.cc:260] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.238890 23786 ts_tablet_manager.cc:1434] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:27.239378 23786 tablet_bootstrap.cc:492] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.239998 24363 ts_tablet_manager.cc:1929] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.236178 24349 raft_consensus.cc:2243] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.245308 24349 raft_consensus.cc:2272] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.246061 24349 ts_tablet_manager.cc:1916] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.249548 23790 tablet_bootstrap.cc:492] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:27.249691 23790 ts_tablet_manager.cc:1403] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.001s
I20251023 20:15:27.250545 23790 raft_consensus.cc:359] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.250695 23790 raft_consensus.cc:385] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.250735 23790 raft_consensus.cc:740] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.251242 24621 raft_consensus.cc:493] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.251358 24621 raft_consensus.cc:515] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.247170 23786 tablet_bootstrap.cc:654] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.251919 24621 leader_election.cc:290] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.252480 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d951134f6773410386bc57f7f90310c3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.253050 23333 leader_election.cc:343] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.255936 24621 consensus_queue.cc:1048] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:27.261937 23333 consensus_peers.cc:597] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edec01a63a3046d9ae5a372fc9068b52. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.262478 24349 ts_tablet_manager.cc:1929] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.262586 24349 log.cc:1199] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/6e1b4748c0244b258960b1025da47562
I20251023 20:15:27.262872 24349 ts_tablet_manager.cc:1950] T 6e1b4748c0244b258960b1025da47562 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.264592 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 84) took 1239 ms. Trace:
I20251023 20:15:27.264683 24349 rpcz_store.cc:276] 1023 20:15:26.025643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.025781 (+ 138us) service_pool.cc:225] Handling call
1023 20:15:27.235983 (+1210202us) ts_tablet_manager.cc:1151] Deleting tablet 6e1b4748c0244b258960b1025da47562
1023 20:15:27.235991 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.254429 (+ 18438us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.260361 (+ 5932us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.264570 (+ 4209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209920}
I20251023 20:15:27.264979 24349 tablet_replica.cc:333] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.265161 24349 raft_consensus.cc:2243] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.265285 24349 raft_consensus.cc:2272] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.265244 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 6e1b4748c0244b258960b1025da47562 (table default.Table_4740fc3fe4ac4927b7f776c980467298 [id=a93a658ff7104725a680f6ee3686e0a4]) successfully deleted
I20251023 20:15:27.265738 24349 ts_tablet_manager.cc:1916] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.270557 23786 tablet_bootstrap.cc:492] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.270747 23786 ts_tablet_manager.cc:1403] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20251023 20:15:27.271432 23786 raft_consensus.cc:359] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.271589 23786 raft_consensus.cc:385] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.271641 23786 raft_consensus.cc:740] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.271827 23786 consensus_queue.cc:260] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.272284 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:27.272604 23786 ts_tablet_manager.cc:1434] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:27.272979 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c317dc31cb0448d985512e6ba98735be (table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]). Expected version 3 got 0
I20251023 20:15:27.273381 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:27.273803 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:27.273981 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:27.274473 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:27.274902 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:27.275192 24349 ts_tablet_manager.cc:1929] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.275274 24349 log.cc:1199] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/4ea63540149c4c02abb8ae51209060a3
I20251023 20:15:27.275565 24349 ts_tablet_manager.cc:1950] T 4ea63540149c4c02abb8ae51209060a3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.279219 24495 consensus_queue.cc:1048] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.281700 23786 tablet_bootstrap.cc:492] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.284160 23786 tablet_bootstrap.cc:654] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.285924 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 85) took 1259 ms. Trace:
I20251023 20:15:27.288691 24349 rpcz_store.cc:276] 1023 20:15:26.026558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.026688 (+ 130us) service_pool.cc:225] Handling call
1023 20:15:27.264935 (+1238247us) ts_tablet_manager.cc:1151] Deleting tablet 4ea63540149c4c02abb8ae51209060a3
1023 20:15:27.264944 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.268586 (+ 3642us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.275081 (+ 6495us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.285906 (+ 10825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237974}
I20251023 20:15:27.289158 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 4ea63540149c4c02abb8ae51209060a3 (table default.Table_e8cec02c09a44acab531a2f71ebb0a73 [id=41b1a0d78a924f2782bcfd5695f835d1]) successfully deleted
I20251023 20:15:27.289534 24349 tablet_replica.cc:333] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.289719 24349 raft_consensus.cc:2243] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.289876 24349 raft_consensus.cc:2272] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.290376 24349 ts_tablet_manager.cc:1916] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.297797 23786 tablet_bootstrap.cc:492] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.297983 23786 ts_tablet_manager.cc:1403] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20251023 20:15:27.298666 23786 raft_consensus.cc:359] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.298862 23786 raft_consensus.cc:385] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.298943 23786 raft_consensus.cc:740] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.299149 23786 consensus_queue.cc:260] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.300379 23786 ts_tablet_manager.cc:1434] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:27.301304 23786 tablet_bootstrap.cc:492] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.302877 23786 tablet_bootstrap.cc:654] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.248373 24655 raft_consensus.cc:493] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.306126 24655 raft_consensus.cc:515] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.306739 24655 leader_election.cc:290] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:27.252514 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d951134f6773410386bc57f7f90310c3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.258740 23787 tablet_bootstrap.cc:492] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.306944 23669 raft_consensus.cc:2468] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.307000 23787 ts_tablet_manager.cc:1403] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.013s
I20251023 20:15:27.307718 23787 raft_consensus.cc:359] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.307869 23787 raft_consensus.cc:385] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.307840 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97e2e0f839a4ef389fe100b6a81301e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.307914 23787 raft_consensus.cc:740] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.308091 23787 consensus_queue.cc:260] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:27.308414 23605 leader_election.cc:343] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.308583 23787 ts_tablet_manager.cc:1434] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251023 20:15:27.308841 23787 tablet_bootstrap.cc:492] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.308944 23333 leader_election.cc:304] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.309396 24670 raft_consensus.cc:2804] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.309465 24670 raft_consensus.cc:493] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.309518 24670 raft_consensus.cc:3060] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.311424 23787 tablet_bootstrap.cc:654] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.311905 24670 raft_consensus.cc:515] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.312395 24670 leader_election.cc:290] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.313534 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d951134f6773410386bc57f7f90310c3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:27.314025 24349 ts_tablet_manager.cc:1929] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.314121 24349 log.cc:1199] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/9d1ddde7e3134e23b3a9356523a93753
I20251023 20:15:27.314464 24349 ts_tablet_manager.cc:1950] T 9d1ddde7e3134e23b3a9356523a93753 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.314407 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d951134f6773410386bc57f7f90310c3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.314561 23669 raft_consensus.cc:3060] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.315121 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 86) took 1288 ms. Trace:
I20251023 20:15:27.315188 24349 rpcz_store.cc:276] 1023 20:15:26.027268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.027308 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:27.289489 (+1262181us) ts_tablet_manager.cc:1151] Deleting tablet 9d1ddde7e3134e23b3a9356523a93753
1023 20:15:27.289498 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.295250 (+ 5752us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.313920 (+ 18670us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.315104 (+ 1184us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1261907}
I20251023 20:15:27.315582 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 9d1ddde7e3134e23b3a9356523a93753 (table default.Table_a1332462902c4ea5971afeb4c3688865 [id=f27defb723394b4db220b278b92e6ae8]) successfully deleted
I20251023 20:15:27.315949 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 87) took 1288 ms. Trace:
I20251023 20:15:27.316013 24349 rpcz_store.cc:276] 1023 20:15:26.027761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.029196 (+ 1435us) service_pool.cc:225] Handling call
1023 20:15:27.315840 (+1286644us) ts_tablet_manager.cc:1151] Deleting tablet 370a790a053446b2b06314c7db242270
1023 20:15:27.315848 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.315933 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286399}
W20251023 20:15:27.316411 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
I20251023 20:15:27.316742 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 88) took 1287 ms. Trace:
I20251023 20:15:27.316864 23669 raft_consensus.cc:2468] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.316814 24349 rpcz_store.cc:276] 1023 20:15:26.029600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.029636 (+ 36us) service_pool.cc:225] Handling call
1023 20:15:27.316633 (+1286997us) ts_tablet_manager.cc:1151] Deleting tablet 1822fdf1f5f44dd4a742aa3c2f8443cf
1023 20:15:27.316641 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.316727 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286809}
W20251023 20:15:27.317201 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
I20251023 20:15:27.317391 23333 leader_election.cc:304] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:27.317490 24349 tablet_replica.cc:333] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.317649 24349 raft_consensus.cc:2243] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:27.317673 23333 leader_election.cc:343] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.317926 24665 raft_consensus.cc:2804] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.317955 24349 raft_consensus.cc:2272] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.318049 24665 raft_consensus.cc:697] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.318262 24665 consensus_queue.cc:237] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.319150 24349 ts_tablet_manager.cc:1916] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.320003 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97e2e0f839a4ef389fe100b6a81301e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.320510 23604 leader_election.cc:343] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.320621 23604 leader_election.cc:304] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.321385 24655 raft_consensus.cc:2749] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.323069 24363 log.cc:1199] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/ae388da9e4f94ba5af82a67088ab7097
I20251023 20:15:27.323462 24363 ts_tablet_manager.cc:1950] T ae388da9e4f94ba5af82a67088ab7097 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:27.324204 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 75) took 1686 ms. Trace:
I20251023 20:15:27.324324 24363 rpcz_store.cc:276] 1023 20:15:25.637514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.638650 (+ 1136us) service_pool.cc:225] Handling call
1023 20:15:27.196941 (+1558291us) ts_tablet_manager.cc:1151] Deleting tablet ae388da9e4f94ba5af82a67088ab7097
1023 20:15:27.196949 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.204868 (+ 7919us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.239873 (+ 35005us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.324188 (+ 84315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1551769}
I20251023 20:15:27.324689 24363 tablet_replica.cc:333] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:27.324851 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet ae388da9e4f94ba5af82a67088ab7097 (table default.Table_7a4278caa64d4373b42c8b865ea6c2d8 [id=8f99b8c48cce4b63af995ef9ddb1edfa]) successfully deleted
I20251023 20:15:27.324894 24363 raft_consensus.cc:2243] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:27.325938 23333 consensus_peers.cc:597] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e1b4748c0244b258960b1025da47562. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.326186 23790 consensus_queue.cc:260] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.329242 24349 ts_tablet_manager.cc:1929] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.329348 24349 log.cc:1199] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/ae388da9e4f94ba5af82a67088ab7097
I20251023 20:15:27.329658 24349 ts_tablet_manager.cc:1950] T ae388da9e4f94ba5af82a67088ab7097 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.330262 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 89) took 1299 ms. Trace:
I20251023 20:15:27.330338 24349 rpcz_store.cc:276] 1023 20:15:26.030324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.030362 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:27.317448 (+1287086us) ts_tablet_manager.cc:1151] Deleting tablet ae388da9e4f94ba5af82a67088ab7097
1023 20:15:27.317457 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.322181 (+ 4724us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.329150 (+ 6969us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.330246 (+ 1096us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286857}
I20251023 20:15:27.330730 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet ae388da9e4f94ba5af82a67088ab7097 (table default.Table_7a4278caa64d4373b42c8b865ea6c2d8 [id=8f99b8c48cce4b63af995ef9ddb1edfa]) successfully deleted
I20251023 20:15:27.333369 23790 ts_tablet_manager.cc:1434] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.084s user 0.000s sys 0.002s
I20251023 20:15:27.333667 23790 tablet_bootstrap.cc:492] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:27.334040 24349 tablet_replica.cc:333] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.334223 24349 raft_consensus.cc:2243] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.334388 24349 raft_consensus.cc:2272] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.337299 24349 ts_tablet_manager.cc:1916] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.339901 23790 tablet_bootstrap.cc:654] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.340945 24363 raft_consensus.cc:2272] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.341614 24363 ts_tablet_manager.cc:1916] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.345901 24615 raft_consensus.cc:493] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.346037 24615 raft_consensus.cc:515] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.346616 24615 leader_election.cc:290] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:27.347664 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa794b3bec4b4b3993c814ea7cd02ee2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.347893 23397 raft_consensus.cc:2393] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:27.347260 23215 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
W20251023 20:15:27.349977 23470 leader_election.cc:343] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: aa794b3bec4b4b3993c814ea7cd02ee2
I20251023 20:15:27.350087 23470 leader_election.cc:304] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.350455 24615 raft_consensus.cc:3060] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.352914 24615 raft_consensus.cc:2749] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.352947 23219 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet af899c01b4af46d39d27f984cc3d5940
I20251023 20:15:27.353515 23219 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 43) took 2190 ms. Trace:
I20251023 20:15:27.353629 23219 rpcz_store.cc:276] 1023 20:15:25.162730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.936053 (+773323us) service_pool.cc:225] Handling call
1023 20:15:25.937017 (+ 964us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:25.937211 (+ 194us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:25.937283 (+ 72us) catalog_manager.cc:2824] Locking tablets
1023 20:15:25.937531 (+ 248us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:26.105431 (+167900us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:26.105529 (+ 98us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:27.352881 (+1247352us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:27.353455 (+ 574us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:25.958166 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:25.958297 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:25.958307 (+ 10us) write_op.cc:435] Acquired schema lock
1023 20:15:25.958310 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:25.958442 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:25.958455 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:25.958459 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:25.958575 (+ 116us) tablet.cc:699] Row locks acquired
1023 20:15:25.958587 (+ 12us) write_op.cc:260] PREPARE: finished
1023 20:15:25.958628 (+ 41us) write_op.cc:270] Start()
1023 20:15:25.958660 (+ 32us) write_op.cc:276] Timestamp: P: 1761250525958575 usec, L: 0
1023 20:15:25.958664 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.084560 (+125896us) log.cc:844] Serialized 765 byte log entry
1023 20:15:26.085384 (+ 824us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 000000000098330a 00007f899878e682 00007f8999b70ca5 00007f8999b6f7c9 00007f8999b77bf0 00007f8999b779fd 0000000000a880ec 00007f898c95576f 00007f898c958a3b 00007f898c9588b8 0000000000a880ec
1023 20:15:26.094364 (+ 8980us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.104393 (+ 10029us) write_op.cc:301] APPLY: starting
1023 20:15:26.104415 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.104465 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.104468 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.104606 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.104644 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:26.104651 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:26.104749 (+ 98us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.105052 (+ 303us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.105316 (+ 264us) write_op.cc:454] Released schema lock
1023 20:15:26.105326 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.105600 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.105605 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9933,"apply.run_cpu_time_us":587,"apply.run_wall_time_us":1205,"num_ops":2,"prepare.queue_time_us":19964,"prepare.run_cpu_time_us":1808,"prepare.run_wall_time_us":127288,"raft.queue_time_us":3596,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":9,"replication_time_us":135667,"spinlock_wait_cycles":288784128}]]}
I20251023 20:15:27.355170 24349 ts_tablet_manager.cc:1929] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.360121 24349 log.cc:1199] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/0438593fbbdf47cebfb64121acad104a
I20251023 20:15:27.360522 24349 ts_tablet_manager.cc:1950] T 0438593fbbdf47cebfb64121acad104a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.361359 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 90) took 1329 ms. Trace:
I20251023 20:15:27.361441 24349 rpcz_store.cc:276] 1023 20:15:26.031498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.031902 (+ 404us) service_pool.cc:225] Handling call
1023 20:15:27.333995 (+1302093us) ts_tablet_manager.cc:1151] Deleting tablet 0438593fbbdf47cebfb64121acad104a
1023 20:15:27.334005 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.340267 (+ 6262us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.355064 (+ 14797us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.361333 (+ 6269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1301873}
I20251023 20:15:27.361985 24349 tablet_replica.cc:333] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.362172 24349 raft_consensus.cc:2243] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.362313 24349 raft_consensus.cc:2272] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.365774 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:27.365880 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 0438593fbbdf47cebfb64121acad104a (table default.Table_e13d2545a0a54b91a8bcb64db45581cc [id=5d8522349b2648fda1e17a6fd1b301dd]) successfully deleted
I20251023 20:15:27.367586 24349 ts_tablet_manager.cc:1916] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.368021 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:27.370592 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:27.378279 24665 raft_consensus.cc:493] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.378471 24665 raft_consensus.cc:515] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.379195 24665 leader_election.cc:290] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.380087 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e08dad1d0f643efb3a821a1b9196290" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.380306 23668 raft_consensus.cc:2468] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.380749 23333 leader_election.cc:304] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:27.381098 24665 raft_consensus.cc:2804] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.381172 24665 raft_consensus.cc:493] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.381225 24665 raft_consensus.cc:3060] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.383757 24665 raft_consensus.cc:515] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.384366 24665 leader_election.cc:290] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:27.385684 23333 consensus_peers.cc:597] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e1b4748c0244b258960b1025da47562. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.385918 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e08dad1d0f643efb3a821a1b9196290" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.386086 23669 raft_consensus.cc:3060] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.388736 23669 raft_consensus.cc:2468] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.389210 23333 leader_election.cc:304] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:27.389513 24665 raft_consensus.cc:2804] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.389631 24665 raft_consensus.cc:697] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.390079 24665 consensus_queue.cc:237] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.399768 24655 consensus_queue.cc:1048] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.400676 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e08dad1d0f643efb3a821a1b9196290" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:27.402050 23333 leader_election.cc:343] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:27.402426 23333 consensus_peers.cc:597] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.403681 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e08dad1d0f643efb3a821a1b9196290" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.404412 23333 leader_election.cc:343] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.412640 24349 ts_tablet_manager.cc:1929] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.412765 24349 log.cc:1199] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/5346a221c78342359eb6e418bd12cd24
I20251023 20:15:27.413071 24349 ts_tablet_manager.cc:1950] T 5346a221c78342359eb6e418bd12cd24 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.413858 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 91) took 1381 ms. Trace:
I20251023 20:15:27.413935 24349 rpcz_store.cc:276] 1023 20:15:26.032727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.032766 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:27.361929 (+1329163us) ts_tablet_manager.cc:1151] Deleting tablet 5346a221c78342359eb6e418bd12cd24
1023 20:15:27.361941 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.374303 (+ 12362us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.412516 (+ 38213us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.413839 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328931}
I20251023 20:15:27.414368 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 5346a221c78342359eb6e418bd12cd24 (table default.Table_54bd013a97df4bb08fdab2eeea9f5d42 [id=71ba16e8b75642e09686eb46c4315366]) successfully deleted
I20251023 20:15:27.414791 24559 consensus_queue.cc:1048] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.415216 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 92) took 1381 ms. Trace:
I20251023 20:15:27.415335 24349 rpcz_store.cc:276] 1023 20:15:26.033711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.033757 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:27.415099 (+1381342us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:27.415108 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.415199 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381138}
W20251023 20:15:27.415846 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
I20251023 20:15:27.416198 24349 tablet_replica.cc:333] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.417294 24349 raft_consensus.cc:2243] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.417557 24349 raft_consensus.cc:2272] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.418180 24349 ts_tablet_manager.cc:1916] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:27.426034 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet af899c01b4af46d39d27f984cc3d5940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af899c01b4af46d39d27f984cc3d5940 already in progress: creating tablet
I20251023 20:15:27.432905 23529 raft_consensus.cc:3060] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.435070 24559 raft_consensus.cc:493] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.435218 24559 raft_consensus.cc:515] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.435850 24559 leader_election.cc:290] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.438374 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.438985 23604 leader_election.cc:343] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.439316 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.439606 24349 ts_tablet_manager.cc:1929] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.439715 24349 log.cc:1199] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a2dbb7997d0f4e6bac48cfb196f5ccb6
I20251023 20:15:27.440062 24349 ts_tablet_manager.cc:1950] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:27.440084 23605 leader_election.cc:343] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.440182 23605 leader_election.cc:304] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.440624 24559 raft_consensus.cc:2749] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.440784 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 93) took 1404 ms. Trace:
I20251023 20:15:27.440871 24349 rpcz_store.cc:276] 1023 20:15:26.036639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.036681 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:27.416146 (+1379465us) ts_tablet_manager.cc:1151] Deleting tablet a2dbb7997d0f4e6bac48cfb196f5ccb6
1023 20:15:27.416157 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.424525 (+ 8368us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.439493 (+ 14968us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.440764 (+ 1271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379279}
I20251023 20:15:27.441563 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 94) took 1402 ms. Trace:
I20251023 20:15:27.441637 24349 rpcz_store.cc:276] 1023 20:15:26.038757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.038797 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:27.441449 (+1402652us) ts_tablet_manager.cc:1151] Deleting tablet 419e41d7a37f4b888cb242b089c28790
1023 20:15:27.441459 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.441547 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402444}
I20251023 20:15:27.442015 24349 tablet_replica.cc:333] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.442181 24349 raft_consensus.cc:2243] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.442351 24349 raft_consensus.cc:2272] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.442658 24559 consensus_queue.cc:1048] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.442869 24349 ts_tablet_manager.cc:1916] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:27.445933 23333 consensus_peers.cc:597] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ea63540149c4c02abb8ae51209060a3. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.454494 24559 raft_consensus.cc:493] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.454640 24559 raft_consensus.cc:515] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.455178 24559 leader_election.cc:290] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.456118 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48436f3d8f1443db990a2d10b668b70a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:27.456647 23605 leader_election.cc:343] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.459335 24349 ts_tablet_manager.cc:1929] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.459452 24349 log.cc:1199] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/52bc660b9da54c5d984381a267e1efa2
I20251023 20:15:27.459793 24349 ts_tablet_manager.cc:1950] T 52bc660b9da54c5d984381a267e1efa2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.460546 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 95) took 1401 ms. Trace:
I20251023 20:15:27.460635 24349 rpcz_store.cc:276] 1023 20:15:26.058684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.058732 (+ 48us) service_pool.cc:225] Handling call
1023 20:15:27.441973 (+1383241us) ts_tablet_manager.cc:1151] Deleting tablet 52bc660b9da54c5d984381a267e1efa2
1023 20:15:27.441983 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.447884 (+ 5901us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.459221 (+ 11337us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.460528 (+ 1307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1382974}
I20251023 20:15:27.461046 24349 tablet_replica.cc:333] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.461242 24349 raft_consensus.cc:2243] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.461380 24349 raft_consensus.cc:2272] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.461869 24349 ts_tablet_manager.cc:1916] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.463776 24553 consensus_queue.cc:1048] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.464119 24553 raft_consensus.cc:493] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.464213 24553 raft_consensus.cc:515] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.464696 24553 leader_election.cc:290] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.466507 24495 consensus_queue.cc:1048] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.467823 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "994a555b24e4432a84601c54d422aa5d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:27.469560 23605 leader_election.cc:343] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.473981 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "994a555b24e4432a84601c54d422aa5d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.476073 24297 consensus_queue.cc:1048] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:27.478120 23604 leader_election.cc:343] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.478274 23604 leader_election.cc:304] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.478295 24349 ts_tablet_manager.cc:1929] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.478394 24349 log.cc:1199] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/33b5efb30b5c491c8eafe3ac25688843
I20251023 20:15:27.478684 24349 ts_tablet_manager.cc:1950] T 33b5efb30b5c491c8eafe3ac25688843 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.478752 24658 raft_consensus.cc:2749] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.479364 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 96) took 1420 ms. Trace:
I20251023 20:15:27.479440 24349 rpcz_store.cc:276] 1023 20:15:26.059145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.059184 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:27.460999 (+1401815us) ts_tablet_manager.cc:1151] Deleting tablet 33b5efb30b5c491c8eafe3ac25688843
1023 20:15:27.461009 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.467176 (+ 6167us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.478185 (+ 11009us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.479346 (+ 1161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401666}
I20251023 20:15:27.480466 24349 tablet_replica.cc:333] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.480638 24349 raft_consensus.cc:2243] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.480770 24349 raft_consensus.cc:2272] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.481292 24349 ts_tablet_manager.cc:1916] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.481859 24665 raft_consensus.cc:493] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.481977 24665 raft_consensus.cc:515] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.482463 24665 leader_election.cc:290] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.483139 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0858358aabff47a69dc77f4669b225c5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.483783 23333 leader_election.cc:343] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:27.484294 23333 leader_election.cc:343] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 0858358aabff47a69dc77f4669b225c5
I20251023 20:15:27.484395 23333 leader_election.cc:304] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.484768 24665 raft_consensus.cc:2749] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.488852 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48436f3d8f1443db990a2d10b668b70a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.489837 23604 leader_election.cc:343] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.489984 23604 leader_election.cc:304] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.490243 24349 ts_tablet_manager.cc:1929] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.490350 24349 log.cc:1199] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/3a4c8dcbdfcf4a15b8a4194ea04b93a0
I20251023 20:15:27.490437 24559 raft_consensus.cc:2749] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.490671 24349 ts_tablet_manager.cc:1950] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.491406 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 97) took 1431 ms. Trace:
I20251023 20:15:27.491519 24349 rpcz_store.cc:276] 1023 20:15:26.059444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.059826 (+ 382us) service_pool.cc:225] Handling call
1023 20:15:27.480420 (+1420594us) ts_tablet_manager.cc:1151] Deleting tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0
1023 20:15:27.480429 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.487331 (+ 6902us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.490129 (+ 2798us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.491360 (+ 1231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420233}
I20251023 20:15:27.499403 24349 tablet_replica.cc:333] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.499624 24349 raft_consensus.cc:2243] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:27.499969 24349 raft_consensus.cc:2272] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.500502 24349 ts_tablet_manager.cc:1916] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.501405 24665 raft_consensus.cc:493] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.501536 24665 raft_consensus.cc:515] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.502074 24665 leader_election.cc:290] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.506063 24349 ts_tablet_manager.cc:1929] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.506021 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.506187 24349 log.cc:1199] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:27.506253 23669 raft_consensus.cc:2468] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.506524 24349 ts_tablet_manager.cc:1950] T 83da09ee0d10435ba89a19dc378d34eb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.506819 23333 leader_election.cc:304] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:27.507157 24665 raft_consensus.cc:2804] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.507243 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 98) took 1446 ms. Trace:
I20251023 20:15:27.507258 24665 raft_consensus.cc:493] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.507323 24665 raft_consensus.cc:3060] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.507321 24349 rpcz_store.cc:276] 1023 20:15:26.060330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.060370 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:27.498275 (+1437905us) ts_tablet_manager.cc:1151] Deleting tablet 83da09ee0d10435ba89a19dc378d34eb
1023 20:15:27.499325 (+ 1050us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.503297 (+ 3972us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.505968 (+ 2671us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.507193 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1437754}
I20251023 20:15:27.509773 24665 raft_consensus.cc:515] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.510347 24665 leader_election.cc:290] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.515803 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:27.515967 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 99) took 1455 ms. Trace:
I20251023 20:15:27.516047 24349 rpcz_store.cc:276] 1023 20:15:26.060646 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.060951 (+ 305us) service_pool.cc:225] Handling call
1023 20:15:27.515798 (+1454847us) ts_tablet_manager.cc:1151] Deleting tablet 01fa5f9c700b458ea147373d78551573
1023 20:15:27.515815 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.515917 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1454633}
I20251023 20:15:27.516438 24349 tablet_replica.cc:333] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.516426 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.516618 24349 raft_consensus.cc:2243] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:27.516635 23669 raft_consensus.cc:3060] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:27.516541 24423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251023 20:15:27.516885 24423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251023 20:15:27.516979 24423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251023 20:15:27.516970 24349 raft_consensus.cc:2272] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.517506 24349 ts_tablet_manager.cc:1916] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.519134 23669 raft_consensus.cc:2468] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.519773 23333 leader_election.cc:304] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:27.520051 23333 leader_election.cc:343] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.520471 24670 raft_consensus.cc:2804] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.520617 24670 raft_consensus.cc:697] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.520867 24670 consensus_queue.cc:237] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.521484 24559 raft_consensus.cc:493] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.521631 24559 raft_consensus.cc:515] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.522197 24559 leader_election.cc:290] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.522889 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:27.523396 23605 leader_election.cc:343] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.523975 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.524456 23604 leader_election.cc:343] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.524591 23604 leader_election.cc:304] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.524993 24559 raft_consensus.cc:2749] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.527455 24665 consensus_queue.cc:1048] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.531021 24349 ts_tablet_manager.cc:1929] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.531154 24349 log.cc:1199] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/cff337a6e728489b99fa8c5d4dfece2b
I20251023 20:15:27.531507 24349 ts_tablet_manager.cc:1950] T cff337a6e728489b99fa8c5d4dfece2b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.532238 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 100) took 1470 ms. Trace:
W20251023 20:15:27.535867 23333 consensus_peers.cc:597] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95574277dde44983bb0a43d2b5ad01d5. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.532339 24349 rpcz_store.cc:276] 1023 20:15:26.061277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.061572 (+ 295us) service_pool.cc:225] Handling call
1023 20:15:27.516365 (+1454793us) ts_tablet_manager.cc:1151] Deleting tablet cff337a6e728489b99fa8c5d4dfece2b
1023 20:15:27.516373 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.527390 (+ 11017us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.530894 (+ 3504us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.532204 (+ 1310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1454625}
I20251023 20:15:27.538244 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.538790 23333 leader_election.cc:343] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.441304 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 (table default.Table_e186a061f84042cd82645b37beae1da1 [id=dc553d161a13467c99df49720181dba1]) successfully deleted
W20251023 20:15:27.539862 23204 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 98548.6us
I20251023 20:15:27.539938 24349 tablet_replica.cc:333] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.540144 24349 raft_consensus.cc:2243] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:27.540133 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 419e41d7a37f4b888cb242b089c28790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 419e41d7a37f4b888cb242b089c28790 already in progress: creating tablet
I20251023 20:15:27.540177 23787 tablet_bootstrap.cc:492] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.540302 24349 raft_consensus.cc:2272] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.540330 23787 ts_tablet_manager.cc:1403] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.232s user 0.022s sys 0.085s
I20251023 20:15:27.540346 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 52bc660b9da54c5d984381a267e1efa2 (table default.Table_0cd3fe0039de476c813272d09a9ba564 [id=161b57ca7cf148899ea058eab98b780d]) successfully deleted
I20251023 20:15:27.540499 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 33b5efb30b5c491c8eafe3ac25688843 (table default.Table_6d625b139835401587825737c53000e3 [id=2ede045acf474fe9a77d38eaed85a15a]) successfully deleted
I20251023 20:15:27.540680 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 (table default.Table_e926815d67914fed915b184838f05b72 [id=265a9707ddea4ee1a19aac6033883822]) successfully deleted
I20251023 20:15:27.540827 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 83da09ee0d10435ba89a19dc378d34eb (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]) successfully deleted
I20251023 20:15:27.540918 24349 ts_tablet_manager.cc:1916] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:27.541033 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
I20251023 20:15:27.541208 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet cff337a6e728489b99fa8c5d4dfece2b (table default.Table_ca4eab8d09574f7abb17e56482816adf [id=1abfa2f299624b41be063c4934d88e64]) successfully deleted
I20251023 20:15:27.541122 23787 raft_consensus.cc:359] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.541311 23787 raft_consensus.cc:385] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.541366 23787 raft_consensus.cc:740] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.541574 23787 consensus_queue.cc:260] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.542021 23787 ts_tablet_manager.cc:1434] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20251023 20:15:27.542330 24423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251023 20:15:27.542413 24423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251023 20:15:27.542493 24423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251023 20:15:27.542560 24423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251023 20:15:27.556455 23790 tablet_bootstrap.cc:492] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:27.556619 23790 ts_tablet_manager.cc:1403] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.223s user 0.001s sys 0.002s
I20251023 20:15:27.557008 23787 tablet_bootstrap.cc:492] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.557734 24349 ts_tablet_manager.cc:1929] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.557922 24349 log.cc:1199] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/3fbad98d3ab24cc5b7726a92fba04cf8
I20251023 20:15:27.558285 24349 ts_tablet_manager.cc:1950] T 3fbad98d3ab24cc5b7726a92fba04cf8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.559073 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 101) took 1497 ms. Trace:
I20251023 20:15:27.559175 24349 rpcz_store.cc:276] 1023 20:15:26.061871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.062530 (+ 659us) service_pool.cc:225] Handling call
1023 20:15:27.537060 (+1474530us) ts_tablet_manager.cc:1151] Deleting tablet 3fbad98d3ab24cc5b7726a92fba04cf8
1023 20:15:27.537069 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.543827 (+ 6758us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.557604 (+ 13777us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.559018 (+ 1414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474348}
I20251023 20:15:27.559595 23787 tablet_bootstrap.cc:654] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.559649 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 3fbad98d3ab24cc5b7726a92fba04cf8 (table default.Table_21f0d4fca62a4309816edd74c6b137bf [id=be3ef152e4fd4281b5caf942c8d404ab]) successfully deleted
I20251023 20:15:27.560089 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 102) took 1497 ms. Trace:
I20251023 20:15:27.560184 24349 rpcz_store.cc:276] 1023 20:15:26.062846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.062885 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:27.559932 (+1497047us) ts_tablet_manager.cc:1151] Deleting tablet d9f73477a06542c0b215ae86716cdc4e
1023 20:15:27.559942 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.560040 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496881}
W20251023 20:15:27.560582 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
I20251023 20:15:27.560884 24349 tablet_replica.cc:333] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.561069 24349 raft_consensus.cc:2243] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.561259 24349 raft_consensus.cc:2272] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.561755 24349 ts_tablet_manager.cc:1916] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.565393 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:27.567236 23786 tablet_bootstrap.cc:492] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.567800 23786 ts_tablet_manager.cc:1403] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.267s user 0.001s sys 0.004s
I20251023 20:15:27.569442 24615 raft_consensus.cc:493] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.569895 24615 raft_consensus.cc:515] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.570465 24615 leader_election.cc:290] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:27.571033 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d82c92e863b2470699cc822b8cb8c14d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.571223 23397 raft_consensus.cc:2468] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:27.571946 23471 leader_election.cc:304] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:27.572216 24615 raft_consensus.cc:2804] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.572327 24615 raft_consensus.cc:493] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.572412 24615 raft_consensus.cc:3060] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:27.573625 23470 leader_election.cc:343] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:27.574797 24615 raft_consensus.cc:515] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.574966 24349 ts_tablet_manager.cc:1929] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.575122 24349 log.cc:1199] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/dafb9db710f94c328186e3179bf0b319
I20251023 20:15:27.576192 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d82c92e863b2470699cc822b8cb8c14d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:27.576346 23397 raft_consensus.cc:3060] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.578514 23790 raft_consensus.cc:359] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.578681 23790 raft_consensus.cc:385] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.578748 23790 raft_consensus.cc:740] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.578866 23397 raft_consensus.cc:2468] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:27.578964 23790 consensus_queue.cc:260] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.581835 24363 ts_tablet_manager.cc:1929] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.590191 23790 ts_tablet_manager.cc:1434] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20251023 20:15:27.590629 23790 tablet_bootstrap.cc:492] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:27.575476 24349 ts_tablet_manager.cc:1950] T dafb9db710f94c328186e3179bf0b319 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.581792 23471 leader_election.cc:304] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:27.591038 24184 raft_consensus.cc:2804] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.591153 24184 raft_consensus.cc:697] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:27.591387 24184 consensus_queue.cc:237] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.597795 24423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.127.148:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.189:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.127.189
--webserver_port=42879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision c56fb2464ecad91a58d4589e1075de20e0f75c71
build type FASTDEBUG
built by None at 23 Oct 2025 20:03:19 UTC on 24a791456cd2
build id 8684
ASAN enabled
W20251023 20:15:27.598587 23470 leader_election.cc:343] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:27.600230 24423 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251023 20:15:27.582266 24615 leader_election.cc:290] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:27.569283 23786 raft_consensus.cc:359] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.598675 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 103) took 1535 ms. Trace:
I20251023 20:15:27.602419 24349 rpcz_store.cc:276] 1023 20:15:26.063259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.063297 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:27.560811 (+1497514us) ts_tablet_manager.cc:1151] Deleting tablet dafb9db710f94c328186e3179bf0b319
1023 20:15:27.560820 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.571711 (+ 10891us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.574838 (+ 3127us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.598635 (+ 23797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497329}
I20251023 20:15:27.602931 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet dafb9db710f94c328186e3179bf0b319 (table default.Table_82f6a78855d1488b9a8085c3bfbebab3 [id=f033b532f86d452c8f14e7f40ead8d62]) successfully deleted
I20251023 20:15:27.603147 24423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251023 20:15:27.603544 23786 raft_consensus.cc:385] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.603654 23786 raft_consensus.cc:740] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.603900 23786 consensus_queue.cc:260] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.604534 23786 ts_tablet_manager.cc:1434] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20251023 20:15:27.604837 23786 tablet_bootstrap.cc:492] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.604876 23790 tablet_bootstrap.cc:654] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.606176 23786 tablet_bootstrap.cc:654] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.606619 24349 tablet_replica.cc:333] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.606782 24349 raft_consensus.cc:2243] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.606947 24349 raft_consensus.cc:2272] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.601156 23787 tablet_bootstrap.cc:492] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.607769 24363 log.cc:1199] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/0438593fbbdf47cebfb64121acad104a
I20251023 20:15:27.608574 24349 ts_tablet_manager.cc:1916] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.607843 23787 ts_tablet_manager.cc:1403] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.004s
I20251023 20:15:27.609710 24363 ts_tablet_manager.cc:1950] T 0438593fbbdf47cebfb64121acad104a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:27.610085 23787 raft_consensus.cc:359] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.615389 23787 raft_consensus.cc:385] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.615526 23787 raft_consensus.cc:740] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.615777 23787 consensus_queue.cc:260] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.616346 23787 ts_tablet_manager.cc:1434] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:27.616694 23787 tablet_bootstrap.cc:492] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.617229 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 76) took 1978 ms. Trace:
I20251023 20:15:27.618562 23787 tablet_bootstrap.cc:654] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.621955 24363 rpcz_store.cc:276] 1023 20:15:25.638906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.639169 (+ 263us) service_pool.cc:225] Handling call
1023 20:15:27.324646 (+1685477us) ts_tablet_manager.cc:1151] Deleting tablet 0438593fbbdf47cebfb64121acad104a
1023 20:15:27.324655 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.344408 (+ 19753us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.581717 (+237309us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.617166 (+ 35449us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"tablet-delete.queue_time_us":1679466}
I20251023 20:15:27.623180 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 0438593fbbdf47cebfb64121acad104a (table default.Table_e13d2545a0a54b91a8bcb64db45581cc [id=5d8522349b2648fda1e17a6fd1b301dd]) successfully deleted
I20251023 20:15:27.635332 23786 tablet_bootstrap.cc:492] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.635507 23786 ts_tablet_manager.cc:1403] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20251023 20:15:27.636300 23786 raft_consensus.cc:359] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.636490 23786 raft_consensus.cc:385] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.636554 23786 raft_consensus.cc:740] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.636792 23786 consensus_queue.cc:260] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.637118 24349 ts_tablet_manager.cc:1929] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.637236 24349 log.cc:1199] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/5964da70989b4e6cb77fe5d0a52ce9cf
I20251023 20:15:27.637578 24363 tablet_replica.cc:333] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:27.637676 23786 ts_tablet_manager.cc:1434] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:27.638646 24363 raft_consensus.cc:2243] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.637974 23786 tablet_bootstrap.cc:492] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.639708 24363 raft_consensus.cc:2272] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.640290 24363 ts_tablet_manager.cc:1916] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.641443 23786 tablet_bootstrap.cc:654] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.645095 23786 tablet_bootstrap.cc:492] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.645220 23786 ts_tablet_manager.cc:1403] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251023 20:15:27.645931 23786 raft_consensus.cc:359] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.646092 23786 raft_consensus.cc:385] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.646148 23786 raft_consensus.cc:740] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.649513 23787 tablet_bootstrap.cc:492] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.649804 23787 ts_tablet_manager.cc:1403] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.004s
I20251023 20:15:27.650553 23787 raft_consensus.cc:359] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.650740 23787 raft_consensus.cc:385] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.650807 23787 raft_consensus.cc:740] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.651069 23787 consensus_queue.cc:260] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.651676 23787 ts_tablet_manager.cc:1434] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:27.651964 23787 tablet_bootstrap.cc:492] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:27.653251 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:27.653854 23787 tablet_bootstrap.cc:654] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.654258 24559 raft_consensus.cc:493] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.654408 24559 raft_consensus.cc:515] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.654486 23366 tablet_service.cc:1505] Processing CreateTablet for tablet a45d01f99d2048d18bdaca7465519030 (DEFAULT_TABLE table=default.Table_9c045162e1824edbbcf0718c49e3b185 [id=b6a25cca1d8e48af8e0791a1e879d6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.654901 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45d01f99d2048d18bdaca7465519030. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.654856 23786 consensus_queue.cc:260] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.654999 24559 leader_election.cc:290] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.655412 23786 ts_tablet_manager.cc:1434] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251023 20:15:27.655994 23510 tablet_service.cc:1505] Processing CreateTablet for tablet a45d01f99d2048d18bdaca7465519030 (DEFAULT_TABLE table=default.Table_9c045162e1824edbbcf0718c49e3b185 [id=b6a25cca1d8e48af8e0791a1e879d6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.656368 23502 tablet_service.cc:1505] Processing CreateTablet for tablet 5a250341262142edb5b747a9e059ee54 (DEFAULT_TABLE table=default.Table_8587355637694585afe381e3e77c3958 [id=59862d7d13674aad83a3df0d07dce998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.656672 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 3b25e1bd1f064b80a938be67961c1f63 (DEFAULT_TABLE table=default.Table_1e09690c826a4c028e998750992ec3d4 [id=f120045598ff41b09f32bbffde0e7092]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.656980 23515 tablet_service.cc:1505] Processing CreateTablet for tablet 139c1bb5ffa743d4a62edf5e98932454 (DEFAULT_TABLE table=default.Table_ef7d0ab458b247c5a1998f0538a7b2ea [id=87bf96dc0f3b4ca6867bb468b7501138]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.657346 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 5a250341262142edb5b747a9e059ee54 (DEFAULT_TABLE table=default.Table_8587355637694585afe381e3e77c3958 [id=59862d7d13674aad83a3df0d07dce998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.657697 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a250341262142edb5b747a9e059ee54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.657841 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fea0afc8f63497d8b5cdf2352a20267" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.658257 23786 tablet_bootstrap.cc:492] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.659098 23646 tablet_service.cc:1505] Processing CreateTablet for tablet a45d01f99d2048d18bdaca7465519030 (DEFAULT_TABLE table=default.Table_9c045162e1824edbbcf0718c49e3b185 [id=b6a25cca1d8e48af8e0791a1e879d6a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.659463 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45d01f99d2048d18bdaca7465519030. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.660396 23786 tablet_bootstrap.cc:654] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.660744 23510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45d01f99d2048d18bdaca7465519030. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.661377 23640 tablet_service.cc:1505] Processing CreateTablet for tablet 5a250341262142edb5b747a9e059ee54 (DEFAULT_TABLE table=default.Table_8587355637694585afe381e3e77c3958 [id=59862d7d13674aad83a3df0d07dce998]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.661716 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a250341262142edb5b747a9e059ee54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.665140 23371 tablet_service.cc:1505] Processing CreateTablet for tablet 3b25e1bd1f064b80a938be67961c1f63 (DEFAULT_TABLE table=default.Table_1e09690c826a4c028e998750992ec3d4 [id=f120045598ff41b09f32bbffde0e7092]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.665339 23634 tablet_service.cc:1505] Processing CreateTablet for tablet 3b25e1bd1f064b80a938be67961c1f63 (DEFAULT_TABLE table=default.Table_1e09690c826a4c028e998750992ec3d4 [id=f120045598ff41b09f32bbffde0e7092]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.665499 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b25e1bd1f064b80a938be67961c1f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.665699 23634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b25e1bd1f064b80a938be67961c1f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.666199 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 139c1bb5ffa743d4a62edf5e98932454 (DEFAULT_TABLE table=default.Table_ef7d0ab458b247c5a1998f0538a7b2ea [id=87bf96dc0f3b4ca6867bb468b7501138]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.666270 24349 ts_tablet_manager.cc:1950] T 5964da70989b4e6cb77fe5d0a52ce9cf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.666517 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 139c1bb5ffa743d4a62edf5e98932454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.667251 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fea0afc8f63497d8b5cdf2352a20267" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.668370 23605 leader_election.cc:343] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:27.668887 23604 leader_election.cc:343] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.669781 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 139c1bb5ffa743d4a62edf5e98932454 (DEFAULT_TABLE table=default.Table_ef7d0ab458b247c5a1998f0538a7b2ea [id=87bf96dc0f3b4ca6867bb468b7501138]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.670029 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 639450a251964b91b9291fdb029c4308 (DEFAULT_TABLE table=default.Table_05604aaaf6604c4c8f929ac257e5824e [id=04588122a9924ac5bcef60dedde8b1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.670509 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 766882a030b446eb9406b3e32d903ee5 (DEFAULT_TABLE table=default.Table_a29ebb47860443d89041e72a86ac92fa [id=12648eb0bb8b4909814e3f2cb9cec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.670995 23372 tablet_service.cc:1505] Processing CreateTablet for tablet abdad7d76eb8441496d38642382f74bb (DEFAULT_TABLE table=default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.669008 23604 leader_election.cc:304] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.671432 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8ded8261ed406d88df394420730a31 (DEFAULT_TABLE table=default.Table_b3b6a3c4f36f4f888a6b5c483002e6ba [id=52a26bb4b9384c829830c1fe988d04be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.671988 24184 raft_consensus.cc:493] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.672102 24184 raft_consensus.cc:515] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.672672 24184 leader_election.cc:290] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:27.673409 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.673662 23669 raft_consensus.cc:2393] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:27.670102 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 139c1bb5ffa743d4a62edf5e98932454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.677413 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8ded8261ed406d88df394420730a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.677888 23635 tablet_service.cc:1505] Processing CreateTablet for tablet 766882a030b446eb9406b3e32d903ee5 (DEFAULT_TABLE table=default.Table_a29ebb47860443d89041e72a86ac92fa [id=12648eb0bb8b4909814e3f2cb9cec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.678128 23637 tablet_service.cc:1505] Processing CreateTablet for tablet 639450a251964b91b9291fdb029c4308 (DEFAULT_TABLE table=default.Table_05604aaaf6604c4c8f929ac257e5824e [id=04588122a9924ac5bcef60dedde8b1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.678316 23635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766882a030b446eb9406b3e32d903ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.678659 23649 tablet_service.cc:1505] Processing CreateTablet for tablet abdad7d76eb8441496d38642382f74bb (DEFAULT_TABLE table=default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.677888 24553 raft_consensus.cc:2749] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.678979 23649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdad7d76eb8441496d38642382f74bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.679512 23631 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8ded8261ed406d88df394420730a31 (DEFAULT_TABLE table=default.Table_b3b6a3c4f36f4f888a6b5c483002e6ba [id=52a26bb4b9384c829830c1fe988d04be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.679804 23631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8ded8261ed406d88df394420730a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.680789 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639450a251964b91b9291fdb029c4308. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.681268 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639450a251964b91b9291fdb029c4308. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.682312 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 639450a251964b91b9291fdb029c4308 (DEFAULT_TABLE table=default.Table_05604aaaf6604c4c8f929ac257e5824e [id=04588122a9924ac5bcef60dedde8b1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.682461 23786 tablet_bootstrap.cc:492] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.682605 23786 ts_tablet_manager.cc:1403] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251023 20:15:27.682693 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 766882a030b446eb9406b3e32d903ee5 (DEFAULT_TABLE table=default.Table_a29ebb47860443d89041e72a86ac92fa [id=12648eb0bb8b4909814e3f2cb9cec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.683046 23514 tablet_service.cc:1505] Processing CreateTablet for tablet abdad7d76eb8441496d38642382f74bb (DEFAULT_TABLE table=default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.683342 23506 tablet_service.cc:1505] Processing CreateTablet for tablet 2d8ded8261ed406d88df394420730a31 (DEFAULT_TABLE table=default.Table_b3b6a3c4f36f4f888a6b5c483002e6ba [id=52a26bb4b9384c829830c1fe988d04be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:27.685516 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdad7d76eb8441496d38642382f74bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.686650 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766882a030b446eb9406b3e32d903ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.687676 24184 raft_consensus.cc:493] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.687817 24184 raft_consensus.cc:515] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.688396 24184 leader_election.cc:290] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:27.691105 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e73ae95c73945f081c298f49d47c904" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:27.692003 23470 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d0e61de7a93d450d9ed79eef82d7e5e3
W20251023 20:15:27.693580 23471 leader_election.cc:343] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:27.693710 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8ded8261ed406d88df394420730a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.693677 23471 leader_election.cc:304] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.694042 23471 leader_election.cc:304] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.694352 24615 raft_consensus.cc:2749] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.694478 24615 raft_consensus.cc:3060] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.695161 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b25e1bd1f064b80a938be67961c1f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.696738 24615 raft_consensus.cc:2749] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.696936 24665 raft_consensus.cc:493] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:27.697471 23604 consensus_peers.cc:597] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.697060 24665 raft_consensus.cc:515] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.698184 24665 leader_election.cc:290] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:27.699106 23333 consensus_peers.cc:597] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52bc660b9da54c5d984381a267e1efa2. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.699923 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.700101 23669 raft_consensus.cc:2468] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.700181 23515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 139c1bb5ffa743d4a62edf5e98932454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.700886 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.701334 23502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a250341262142edb5b747a9e059ee54. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:27.701478 23333 leader_election.cc:343] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.701583 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766882a030b446eb9406b3e32d903ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.701664 23333 leader_election.cc:304] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.702142 24665 raft_consensus.cc:2804] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.702211 24665 raft_consensus.cc:493] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.702263 24665 raft_consensus.cc:3060] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.704663 24665 raft_consensus.cc:515] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.706887 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdad7d76eb8441496d38642382f74bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.712518 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639450a251964b91b9291fdb029c4308. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:27.712886 23787 tablet_bootstrap.cc:492] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.713008 23787 ts_tablet_manager.cc:1403] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.003s
I20251023 20:15:27.713744 23787 raft_consensus.cc:359] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.713894 23787 raft_consensus.cc:385] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.713934 23787 raft_consensus.cc:740] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.714104 23787 consensus_queue.cc:260] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.714223 24665 leader_election.cc:290] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.714577 23787 ts_tablet_manager.cc:1434] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:27.714769 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.714874 24553 raft_consensus.cc:493] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.714920 23669 raft_consensus.cc:3060] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.714970 24553 raft_consensus.cc:515] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.715540 24553 leader_election.cc:290] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.716054 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.716516 23604 leader_election.cc:343] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.717245 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.717356 23669 raft_consensus.cc:2468] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:27.717734 23605 leader_election.cc:343] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.717845 23605 leader_election.cc:304] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.717914 23333 leader_election.cc:304] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:27.718212 24559 raft_consensus.cc:2749] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.718403 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:27.718942 23333 leader_election.cc:343] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.719296 24665 raft_consensus.cc:493] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:27.719458 24665 raft_consensus.cc:515] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.720072 24665 leader_election.cc:290] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.721167 24495 raft_consensus.cc:2804] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.721601 24495 raft_consensus.cc:697] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.721832 24495 consensus_queue.cc:237] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:27.721446 23333 leader_election.cc:343] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: f4778436aed64b53adb2bab54bcf263b
W20251023 20:15:27.724712 23333 leader_election.cc:343] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: f4778436aed64b53adb2bab54bcf263b
I20251023 20:15:27.725036 23333 leader_election.cc:304] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.725517 24621 raft_consensus.cc:2749] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:27.737653 24184 raft_consensus.cc:493] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.737823 24184 raft_consensus.cc:515] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.741381 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:27.743237 24184 leader_election.cc:290] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:27.729036 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
W20251023 20:15:27.746593 23471 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 3894dd4feeea4c16861a6c9520059474
W20251023 20:15:27.746735 23470 leader_election.cc:343] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3894dd4feeea4c16861a6c9520059474
I20251023 20:15:27.746817 23470 leader_election.cc:304] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:27.747061 24184 raft_consensus.cc:2749] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:27.758008 24746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251023 20:15:27.762650 23787 tablet_bootstrap.cc:492] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.764215 23787 tablet_bootstrap.cc:654] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.683352 23786 raft_consensus.cc:359] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.764621 23786 raft_consensus.cc:385] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.764689 23786 raft_consensus.cc:740] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.764870 23786 consensus_queue.cc:260] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.764884 24621 consensus_queue.cc:1048] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.765861 23786 ts_tablet_manager.cc:1434] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.083s user 0.002s sys 0.000s
I20251023 20:15:27.766146 23786 tablet_bootstrap.cc:492] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.767776 23786 tablet_bootstrap.cc:654] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.769857 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 104) took 1706 ms. Trace:
I20251023 20:15:27.769950 24349 rpcz_store.cc:276] 1023 20:15:26.063588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.063625 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:27.606552 (+1542927us) ts_tablet_manager.cc:1151] Deleting tablet 5964da70989b4e6cb77fe5d0a52ce9cf
1023 20:15:27.606562 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.619639 (+ 13077us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.636983 (+ 17344us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.769838 (+132855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542780}
I20251023 20:15:27.770367 24349 tablet_replica.cc:333] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.770363 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 5964da70989b4e6cb77fe5d0a52ce9cf (table default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]) successfully deleted
I20251023 20:15:27.770548 24349 raft_consensus.cc:2243] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.770704 24349 raft_consensus.cc:2272] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.771205 24349 ts_tablet_manager.cc:1916] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:27.771211 24751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251023 20:15:27.773387 23333 consensus_peers.cc:597] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.774772 23790 tablet_bootstrap.cc:492] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:27.774904 23790 ts_tablet_manager.cc:1403] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.184s user 0.005s sys 0.001s
I20251023 20:15:27.775738 23790 raft_consensus.cc:359] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.775910 23790 raft_consensus.cc:385] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.775964 23790 raft_consensus.cc:740] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.776129 23790 consensus_queue.cc:260] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.776607 23790 ts_tablet_manager.cc:1434] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:27.776862 23790 tablet_bootstrap.cc:492] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:27.777835 23787 tablet_bootstrap.cc:492] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.778012 23787 ts_tablet_manager.cc:1403] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20251023 20:15:27.778563 23790 tablet_bootstrap.cc:654] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.778838 23787 raft_consensus.cc:359] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.779057 23787 raft_consensus.cc:385] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.779142 23787 raft_consensus.cc:740] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.779345 23787 consensus_queue.cc:260] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.780237 23787 ts_tablet_manager.cc:1434] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:27.780587 23787 tablet_bootstrap.cc:492] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.782233 23787 tablet_bootstrap.cc:654] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:27.784821 23333 consensus_peers.cc:597] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.790686 24559 consensus_queue.cc:1048] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.798019 23787 tablet_bootstrap.cc:492] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.798161 23787 ts_tablet_manager.cc:1403] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.010s
I20251023 20:15:27.798856 23787 raft_consensus.cc:359] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.799013 23787 raft_consensus.cc:385] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.799057 23787 raft_consensus.cc:740] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.799256 23787 consensus_queue.cc:260] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.800662 23787 ts_tablet_manager.cc:1434] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:27.801066 23787 tablet_bootstrap.cc:492] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.802971 23787 tablet_bootstrap.cc:654] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.810513 24363 ts_tablet_manager.cc:1929] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.822234 24423 server_base.cc:1047] running on GCE node
I20251023 20:15:27.823601 24423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251023 20:15:27.827936 23790 tablet_bootstrap.cc:492] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:27.828821 23790 ts_tablet_manager.cc:1403] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.001s
I20251023 20:15:27.829936 23790 raft_consensus.cc:359] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.831166 23790 raft_consensus.cc:385] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.831357 23790 raft_consensus.cc:740] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.831782 23790 consensus_queue.cc:260] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.833674 23790 ts_tablet_manager.cc:1434] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:27.834127 23790 tablet_bootstrap.cc:492] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:27.838562 23786 tablet_bootstrap.cc:492] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.842468 23790 tablet_bootstrap.cc:654] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.842737 23786 ts_tablet_manager.cc:1403] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.001s
I20251023 20:15:27.850625 24349 ts_tablet_manager.cc:1929] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.810734 23787 tablet_bootstrap.cc:492] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.851413 23787 ts_tablet_manager.cc:1403] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.009s
I20251023 20:15:27.850800 23786 raft_consensus.cc:359] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.851625 23786 raft_consensus.cc:385] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.851727 23786 raft_consensus.cc:740] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.851279 24363 log.cc:1199] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/5346a221c78342359eb6e418bd12cd24
I20251023 20:15:27.852324 24349 log.cc:1199] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/3894dd4feeea4c16861a6c9520059474
I20251023 20:15:27.852113 23787 raft_consensus.cc:359] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:27.852522 23604 consensus_peers.cc:597] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.852557 23787 raft_consensus.cc:385] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.852613 23787 raft_consensus.cc:740] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.852874 24363 ts_tablet_manager.cc:1950] T 5346a221c78342359eb6e418bd12cd24 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:27.853384 24349 ts_tablet_manager.cc:1950] T 3894dd4feeea4c16861a6c9520059474 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.852793 23787 consensus_queue.cc:260] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.855314 23787 ts_tablet_manager.cc:1434] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251023 20:15:27.855576 23787 tablet_bootstrap.cc:492] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.853567 23786 consensus_queue.cc:260] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.858037 23786 ts_tablet_manager.cc:1434] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251023 20:15:27.858927 24655 raft_consensus.cc:493] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.859040 24655 raft_consensus.cc:515] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.859478 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 77) took 2220 ms. Trace:
I20251023 20:15:27.859557 24655 leader_election.cc:290] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.860577 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "600f0110f8bd4b05983d5922d871c266" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.861058 23786 tablet_bootstrap.cc:492] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:27.861456 23604 leader_election.cc:343] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.858847 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 105) took 1794 ms. Trace:
I20251023 20:15:27.864112 24349 rpcz_store.cc:276] 1023 20:15:26.064497 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.097172 (+ 32675us) service_pool.cc:225] Handling call
1023 20:15:27.770319 (+1673147us) ts_tablet_manager.cc:1151] Deleting tablet 3894dd4feeea4c16861a6c9520059474
1023 20:15:27.770328 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.796093 (+ 25765us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.840281 (+ 44188us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.858829 (+ 18548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672855}
I20251023 20:15:27.864465 24349 tablet_replica.cc:333] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.864657 24349 raft_consensus.cc:2243] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.864828 24349 raft_consensus.cc:2272] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.865008 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 3894dd4feeea4c16861a6c9520059474 (table default.Table_67a76ff63d4d411582ab8e10ac4f6924 [id=fb6dc9a69da44f35a0184490d90a1e5c]) successfully deleted
I20251023 20:15:27.865398 24349 ts_tablet_manager.cc:1916] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.865568 23786 tablet_bootstrap.cc:654] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.871165 24349 ts_tablet_manager.cc:1929] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.871282 24349 log.cc:1199] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e0f8d8184a68453abd2792ccc7740aa1
I20251023 20:15:27.871292 24495 raft_consensus.cc:493] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.871546 24349 ts_tablet_manager.cc:1950] T e0f8d8184a68453abd2792ccc7740aa1 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.872182 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 107) took 1773 ms. Trace:
I20251023 20:15:27.872262 24349 rpcz_store.cc:276] 1023 20:15:26.098725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.099763 (+ 1038us) service_pool.cc:225] Handling call
1023 20:15:27.864419 (+1764656us) ts_tablet_manager.cc:1151] Deleting tablet e0f8d8184a68453abd2792ccc7740aa1
1023 20:15:27.864430 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.868285 (+ 3855us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.871074 (+ 2789us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.872164 (+ 1090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764370}
I20251023 20:15:27.872565 24349 tablet_replica.cc:333] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.872731 24349 raft_consensus.cc:2243] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.872879 24349 raft_consensus.cc:2272] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.871503 24495 raft_consensus.cc:515] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.873390 24349 ts_tablet_manager.cc:1916] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.873796 23787 tablet_bootstrap.cc:654] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.874428 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a64a7c2e1f242e1b54243489622bae5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:27.874596 23669 raft_consensus.cc:2468] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:27.875638 23532 raft_consensus.cc:3060] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.876739 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet e0f8d8184a68453abd2792ccc7740aa1 (table default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]) successfully deleted
I20251023 20:15:27.877655 23787 tablet_bootstrap.cc:492] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:27.877766 23787 ts_tablet_manager.cc:1403] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251023 20:15:27.878715 23787 raft_consensus.cc:359] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.879251 23787 raft_consensus.cc:385] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.879424 23787 raft_consensus.cc:740] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.879513 23333 leader_election.cc:304] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:27.879611 24349 ts_tablet_manager.cc:1929] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.879611 23787 consensus_queue.cc:260] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:27.879712 24349 log.cc:1199] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/7645c36c983f438f8b1223256fd3ad2c
I20251023 20:15:27.880077 24349 ts_tablet_manager.cc:1950] T 7645c36c983f438f8b1223256fd3ad2c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.880074 23787 ts_tablet_manager.cc:1434] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:27.880337 24621 raft_consensus.cc:2804] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:27.880442 24621 raft_consensus.cc:493] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:27.880530 24621 raft_consensus.cc:3060] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.880805 23787 tablet_bootstrap.cc:492] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:27.880780 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 108) took 1780 ms. Trace:
I20251023 20:15:27.880870 24349 rpcz_store.cc:276] 1023 20:15:26.100199 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.101499 (+ 1300us) service_pool.cc:225] Handling call
1023 20:15:27.872516 (+1771017us) ts_tablet_manager.cc:1151] Deleting tablet 7645c36c983f438f8b1223256fd3ad2c
1023 20:15:27.872527 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.876413 (+ 3886us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.879516 (+ 3103us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.880757 (+ 1241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1770584}
I20251023 20:15:27.881202 24349 tablet_replica.cc:333] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.881413 24349 raft_consensus.cc:2243] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.881574 24349 raft_consensus.cc:2272] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.881805 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 7645c36c983f438f8b1223256fd3ad2c (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]) successfully deleted
I20251023 20:15:27.882087 24349 ts_tablet_manager.cc:1916] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.883021 23787 tablet_bootstrap.cc:654] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.885661 24621 raft_consensus.cc:515] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.886325 24621 leader_election.cc:290] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.887164 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a64a7c2e1f242e1b54243489622bae5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:27.887336 23665 raft_consensus.cc:3060] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.887750 24349 ts_tablet_manager.cc:1929] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.887854 24349 log.cc:1199] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:27.888170 24349 ts_tablet_manager.cc:1950] T 66a2f3adb5d347a0adc198cd1ef93304 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.888875 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 109) took 1786 ms. Trace:
I20251023 20:15:27.888950 24349 rpcz_store.cc:276] 1023 20:15:26.102820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.102885 (+ 65us) service_pool.cc:225] Handling call
1023 20:15:27.881148 (+1778263us) ts_tablet_manager.cc:1151] Deleting tablet 66a2f3adb5d347a0adc198cd1ef93304
1023 20:15:27.881159 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.884861 (+ 3702us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.887648 (+ 2787us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.888849 (+ 1201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1777889}
I20251023 20:15:27.889380 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 66a2f3adb5d347a0adc198cd1ef93304 (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]) successfully deleted
I20251023 20:15:27.889467 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 110) took 1786 ms. Trace:
I20251023 20:15:27.889524 24349 rpcz_store.cc:276] 1023 20:15:26.103499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.104387 (+ 888us) service_pool.cc:225] Handling call
1023 20:15:27.889337 (+1784950us) ts_tablet_manager.cc:1151] Deleting tablet 0858358aabff47a69dc77f4669b225c5
1023 20:15:27.889347 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.889450 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1784664}
I20251023 20:15:27.889917 24349 tablet_replica.cc:333] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:27.889887 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 0858358aabff47a69dc77f4669b225c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0858358aabff47a69dc77f4669b225c5 already in progress: creating tablet
I20251023 20:15:27.889895 23665 raft_consensus.cc:2468] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:27.890087 24349 raft_consensus.cc:2243] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:27.890405 24349 raft_consensus.cc:2272] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.890873 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a64a7c2e1f242e1b54243489622bae5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:27.891250 24349 ts_tablet_manager.cc:1916] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:27.891690 23333 leader_election.cc:343] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.891943 23333 leader_election.cc:304] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.892506 24670 raft_consensus.cc:2804] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:27.892620 24670 raft_consensus.cc:697] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:27.892859 24670 consensus_queue.cc:237] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.893414 23786 tablet_bootstrap.cc:492] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:27.893539 23786 ts_tablet_manager.cc:1403] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.007s
I20251023 20:15:27.893954 24665 consensus_queue.cc:1048] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.894212 23786 raft_consensus.cc:359] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.894354 23786 raft_consensus.cc:385] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:27.894402 23786 raft_consensus.cc:740] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:27.897158 24349 ts_tablet_manager.cc:1929] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.897253 24349 log.cc:1199] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e418a62d00824cff89b37a7bace5c469
I20251023 20:15:27.897540 24349 ts_tablet_manager.cc:1950] T e418a62d00824cff89b37a7bace5c469 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.898065 23786 consensus_queue.cc:260] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:27.898162 23529 raft_consensus.cc:3060] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.898195 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 111) took 1793 ms. Trace:
I20251023 20:15:27.898262 24349 rpcz_store.cc:276] 1023 20:15:26.104812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.107966 (+ 3154us) service_pool.cc:225] Handling call
1023 20:15:27.889876 (+1781910us) ts_tablet_manager.cc:1151] Deleting tablet e418a62d00824cff89b37a7bace5c469
1023 20:15:27.889885 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.894243 (+ 4358us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.897017 (+ 2774us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.898180 (+ 1163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1781606}
I20251023 20:15:27.898988 23786 ts_tablet_manager.cc:1434] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251023 20:15:27.899084 24349 tablet_replica.cc:333] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.899235 23786 tablet_bootstrap.cc:492] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:27.899744 24349 raft_consensus.cc:2243] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.900064 24349 raft_consensus.cc:2272] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.900936 24349 ts_tablet_manager.cc:1916] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.901122 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet e418a62d00824cff89b37a7bace5c469 (table default.Table_3a893d487d5b46ca84e6d00ce6cf7137 [id=751ae1aba5564ffa8dd6274778d1b8c7]) successfully deleted
I20251023 20:15:27.901675 24665 consensus_queue.cc:1048] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:27.903970 23333 consensus_peers.cc:597] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d1ddde7e3134e23b3a9356523a93753. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.904029 23786 tablet_bootstrap.cc:654] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:27.907742 24349 ts_tablet_manager.cc:1929] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.907853 24349 log.cc:1199] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/86b9f4c5d9e0469c87071a33722ce3eb
I20251023 20:15:27.908201 24349 ts_tablet_manager.cc:1950] T 86b9f4c5d9e0469c87071a33722ce3eb P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:27.908939 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 112) took 1800 ms. Trace:
I20251023 20:15:27.909034 24349 rpcz_store.cc:276] 1023 20:15:26.108446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.110259 (+ 1813us) service_pool.cc:225] Handling call
1023 20:15:27.898612 (+1788353us) ts_tablet_manager.cc:1151] Deleting tablet 86b9f4c5d9e0469c87071a33722ce3eb
1023 20:15:27.898620 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.904703 (+ 6083us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.907655 (+ 2952us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.908918 (+ 1263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1788095}
I20251023 20:15:27.909456 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 86b9f4c5d9e0469c87071a33722ce3eb (table default.Table_b3b5199d1faf4e5795ba391229812ea9 [id=8ec2443a50674a67a3393236ee4bc48e]) successfully deleted
I20251023 20:15:27.909554 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 113) took 1798 ms. Trace:
I20251023 20:15:27.909612 24349 rpcz_store.cc:276] 1023 20:15:26.110665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.111104 (+ 439us) service_pool.cc:225] Handling call
1023 20:15:27.909433 (+1798329us) ts_tablet_manager.cc:1151] Deleting tablet db758a7a7a89410f8ac1da72048fbc56
1023 20:15:27.909443 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.909538 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1797675}
I20251023 20:15:27.909971 24349 tablet_replica.cc:333] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:27.909986 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
I20251023 20:15:27.910137 24349 raft_consensus.cc:2243] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.910295 24349 raft_consensus.cc:2272] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.911630 24349 ts_tablet_manager.cc:1916] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.917071 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a64a7c2e1f242e1b54243489622bae5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.918061 24349 ts_tablet_manager.cc:1929] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:27.918152 24349 log.cc:1199] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/1e8eb7603c3e46f1b15cdff7a7bbedcf
I20251023 20:15:27.918462 24349 ts_tablet_manager.cc:1950] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:27.917582 23333 leader_election.cc:343] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.917066 24495 leader_election.cc:290] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.919117 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 114) took 1807 ms. Trace:
I20251023 20:15:27.919188 24349 rpcz_store.cc:276] 1023 20:15:26.111885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.112302 (+ 417us) service_pool.cc:225] Handling call
1023 20:15:27.909929 (+1797627us) ts_tablet_manager.cc:1151] Deleting tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf
1023 20:15:27.909938 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.915262 (+ 5324us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.917977 (+ 2715us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.919100 (+ 1123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1797399}
I20251023 20:15:27.919634 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf (table default.Table_1fd972f3964245fab674e2cfc26a8210 [id=cf5e426caddf4508bc0780b9809fff5e]) successfully deleted
I20251023 20:15:27.922780 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "600f0110f8bd4b05983d5922d871c266" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:27.923302 23605 leader_election.cc:343] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.923434 23605 leader_election.cc:304] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.923801 24655 raft_consensus.cc:2749] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.931835 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:27.933513 24655 raft_consensus.cc:493] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.933663 24655 raft_consensus.cc:515] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.934260 24655 leader_election.cc:290] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.934890 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a636a4cdeb404e9f806d8e166ea57938" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:27.935353 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a636a4cdeb404e9f806d8e166ea57938" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.935365 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
W20251023 20:15:27.935472 23604 leader_election.cc:343] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.935573 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
W20251023 20:15:27.935883 23605 leader_election.cc:343] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.935986 23605 leader_election.cc:304] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.936173 24655 raft_consensus.cc:2749] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.936323 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:27.937287 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:27.937605 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:27.937865 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:27.938200 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:27.938601 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:27.938745 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:27.938906 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:27.939069 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:27.939213 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:27.939431 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:27.939646 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:27.939828 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:27.940160 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:27.940297 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:27.940491 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:27.941207 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 115) took 1828 ms. Trace:
I20251023 20:15:27.941318 24349 rpcz_store.cc:276] 1023 20:15:26.112679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.114649 (+ 1970us) service_pool.cc:225] Handling call
1023 20:15:27.941008 (+1826359us) ts_tablet_manager.cc:1151] Deleting tablet 15ced74d0dd74e5195ed98358957d0dd
1023 20:15:27.941022 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.941184 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1826132}
W20251023 20:15:27.941763 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 15ced74d0dd74e5195ed98358957d0dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15ced74d0dd74e5195ed98358957d0dd already in progress: creating tablet
I20251023 20:15:27.942241 24423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251023 20:15:27.943205 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 116) took 1828 ms. Trace:
I20251023 20:15:27.943627 24349 rpcz_store.cc:276] 1023 20:15:26.115023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.115562 (+ 539us) service_pool.cc:225] Handling call
1023 20:15:27.943091 (+1827529us) ts_tablet_manager.cc:1151] Deleting tablet 252b5480222f49439052c86a4115cc9b
1023 20:15:27.943101 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.943189 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1827287}
W20251023 20:15:27.944248 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 252b5480222f49439052c86a4115cc9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 252b5480222f49439052c86a4115cc9b already in progress: creating tablet
I20251023 20:15:27.945190 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 117) took 1829 ms. Trace:
I20251023 20:15:27.945453 24349 rpcz_store.cc:276] 1023 20:15:26.115990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.116734 (+ 744us) service_pool.cc:225] Handling call
1023 20:15:27.944094 (+1827360us) ts_tablet_manager.cc:1151] Deleting tablet 5acdc66df9c748a1a6300962f62fcf36
1023 20:15:27.944574 (+ 480us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.945173 (+ 599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1827138}
W20251023 20:15:27.946076 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
I20251023 20:15:27.947335 24349 tablet_replica.cc:333] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.947657 24349 raft_consensus.cc:2243] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.948052 24349 raft_consensus.cc:2272] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.948788 24349 ts_tablet_manager.cc:1916] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.956243 23531 raft_consensus.cc:3060] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:27.958280 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:27.958519 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:27.958700 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:27.958925 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:27.959410 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:27.964999 24349 ts_tablet_manager.cc:1929] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:27.965158 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:27.965509 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:27.967445 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:27.967618 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:27.967938 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]). Expected version 1 got 0
I20251023 20:15:27.968106 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:27.968304 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:27.968452 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:27.968632 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:27.968878 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:27.969054 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:27.973923 24423 hybrid_clock.cc:648] HybridClock initialized: now 1761250527973788 us; error 120 us; skew 500 ppm
I20251023 20:15:27.974416 24423 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251023 20:15:27.975729 24349 log.cc:1199] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/23225d31c2e647398aa58494c627507e
I20251023 20:15:27.976649 23531 raft_consensus.cc:3060] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:27.980597 23604 consensus_peers.cc:597] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66a2f3adb5d347a0adc198cd1ef93304. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:27.980814 24621 consensus_queue.cc:1048] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.984977 24659 raft_consensus.cc:493] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:27.985122 24659 raft_consensus.cc:515] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:27.985621 24659 leader_election.cc:290] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:27.986370 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:27.986646 24349 ts_tablet_manager.cc:1950] T 23225d31c2e647398aa58494c627507e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:27.986902 23605 leader_election.cc:343] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.987318 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:27.987852 23604 leader_election.cc:343] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:27.987952 23604 leader_election.cc:304] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:27.988435 24659 raft_consensus.cc:2749] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:27.988533 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:27.987340 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 118) took 1870 ms. Trace:
I20251023 20:15:27.990837 24349 rpcz_store.cc:276] 1023 20:15:26.117101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.117972 (+ 871us) service_pool.cc:225] Handling call
1023 20:15:27.947283 (+1829311us) ts_tablet_manager.cc:1151] Deleting tablet 23225d31c2e647398aa58494c627507e
1023 20:15:27.947297 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.953059 (+ 5762us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.964896 (+ 11837us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.987321 (+ 22425us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1827594}
I20251023 20:15:27.991694 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 23225d31c2e647398aa58494c627507e (table default.Table_67a7910474894d659750844c77625a9d [id=82c65a189d13499c8471cb1ce0894c45]) successfully deleted
I20251023 20:15:27.992205 24349 tablet_replica.cc:333] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:27.992601 24349 raft_consensus.cc:2243] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:27.992826 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:27.993270 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:27.993538 24349 raft_consensus.cc:2272] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:27.994377 24349 ts_tablet_manager.cc:1916] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:27.994750 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:27.995713 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8]). Expected version 1 got 0
I20251023 20:15:28.001415 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:28.001874 24659 raft_consensus.cc:493] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.002022 24659 raft_consensus.cc:515] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.002609 24659 leader_election.cc:290] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.003054 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:28.003472 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f103bece2fa240c19c23ac51046c13ed" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.003679 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
W20251023 20:15:28.004132 23605 leader_election.cc:343] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.004439 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f103bece2fa240c19c23ac51046c13ed" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:28.004904 23604 leader_election.cc:343] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.005000 23604 leader_election.cc:304] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.005255 23218 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 9a4f012469fa4e8e827e485287ad76dd
I20251023 20:15:28.005496 24659 raft_consensus.cc:2749] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.005726 23218 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 45) took 2842 ms. Trace:
I20251023 20:15:28.005824 23218 rpcz_store.cc:276] 1023 20:15:25.162904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.187050 (+1024146us) service_pool.cc:225] Handling call
1023 20:15:26.187563 (+ 513us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:26.187723 (+ 160us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:26.187793 (+ 70us) catalog_manager.cc:2824] Locking tablets
1023 20:15:26.357318 (+169525us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:26.368713 (+ 11395us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:26.368800 (+ 87us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:28.005210 (+1636410us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:28.005677 (+ 467us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.361275 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.361409 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.361416 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:26.361419 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:26.361543 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.361556 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.361560 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.361686 (+ 126us) tablet.cc:699] Row locks acquired
1023 20:15:26.361689 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:26.361727 (+ 38us) write_op.cc:270] Start()
1023 20:15:26.361764 (+ 37us) write_op.cc:276] Timestamp: P: 1761250526361686 usec, L: 0
1023 20:15:26.361768 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.361968 (+ 200us) log.cc:844] Serialized 734 byte log entry
1023 20:15:26.367724 (+ 5756us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.367850 (+ 126us) write_op.cc:301] APPLY: starting
1023 20:15:26.367868 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.367918 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.367955 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.368027 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.368062 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:26.368070 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:26.368153 (+ 83us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.368220 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.368243 (+ 23us) write_op.cc:454] Released schema lock
1023 20:15:26.368251 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.368317 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.368320 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":460,"num_ops":2,"prepare.queue_time_us":3194,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":826,"raft.queue_time_us":182,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":233,"replication_time_us":5925}]]}
W20251023 20:15:28.006714 23218 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 192) took 1286 ms (client timeout 1000 ms). Trace:
W20251023 20:15:28.006804 23218 rpcz_store.cc:269] 1023 20:15:26.720630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.006461 (+1285831us) service_pool.cc:212] Skipping call since client already timed out
1023 20:15:28.006669 (+ 208us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 20:15:28.008926 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.009182 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.009255 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:28.011206 23333 consensus_peers.cc:597] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4c8dcbdfcf4a15b8a4194ea04b93a0. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.022326 24659 raft_consensus.cc:493] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.022476 24659 raft_consensus.cc:515] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.023070 24659 leader_election.cc:290] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.024209 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af899c01b4af46d39d27f984cc3d5940" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:28.024857 23605 leader_election.cc:343] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.025354 24659 raft_consensus.cc:493] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.025468 24659 raft_consensus.cc:515] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.026012 24659 leader_election.cc:290] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.026652 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ae910ac41fd4e349f159ab27e44375a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.026988 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ae910ac41fd4e349f159ab27e44375a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.027158 23395 raft_consensus.cc:2468] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
W20251023 20:15:28.027171 23604 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.027500 23605 leader_election.cc:304] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.028173 24658 raft_consensus.cc:2804] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.028247 24658 raft_consensus.cc:493] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.028306 24658 raft_consensus.cc:3060] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.031090 24658 raft_consensus.cc:515] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.032992 24658 leader_election.cc:290] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.033387 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ae910ac41fd4e349f159ab27e44375a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.033515 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ae910ac41fd4e349f159ab27e44375a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:28.033690 23395 raft_consensus.cc:3060] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.034741 24349 ts_tablet_manager.cc:1929] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.034871 24349 log.cc:1199] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:28.035125 24349 ts_tablet_manager.cc:1950] T d0e61de7a93d450d9ed79eef82d7e5e3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.035779 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 119) took 1917 ms. Trace:
I20251023 20:15:28.035871 24349 rpcz_store.cc:276] 1023 20:15:26.118513 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.118978 (+ 465us) service_pool.cc:225] Handling call
1023 20:15:27.991559 (+1872581us) ts_tablet_manager.cc:1151] Deleting tablet d0e61de7a93d450d9ed79eef82d7e5e3
1023 20:15:27.992166 (+ 607us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.999003 (+ 6837us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.034601 (+ 35598us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.035762 (+ 1161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872335}
I20251023 20:15:28.036289 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d0e61de7a93d450d9ed79eef82d7e5e3 (table default.Table_4d358925f1254129a8b42bfca11f8870 [id=a1701c81a217439197ef53ff39675ef9]) successfully deleted
I20251023 20:15:28.036425 23395 raft_consensus.cc:2468] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:28.037465 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af899c01b4af46d39d27f984cc3d5940" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:28.033947 23604 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:28.038161 23604 leader_election.cc:343] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.038264 23604 leader_election.cc:304] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.038858 23605 leader_election.cc:304] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.039062 24659 raft_consensus.cc:2749] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.039175 24658 raft_consensus.cc:2804] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
W20251023 20:15:28.039193 23206 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2a27706ecdb4ab78bb96c10a9f6757f consensus state has no leader
I20251023 20:15:28.039268 24658 raft_consensus.cc:697] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:28.039482 24658 consensus_queue.cc:237] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.045567 23787 tablet_bootstrap.cc:492] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.045699 23787 ts_tablet_manager.cc:1403] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.165s user 0.010s sys 0.035s
I20251023 20:15:28.045722 24495 consensus_queue.cc:1048] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.046435 23787 raft_consensus.cc:359] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.046588 23787 raft_consensus.cc:385] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.046648 23787 raft_consensus.cc:740] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.046840 23787 consensus_queue.cc:260] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.048650 24495 consensus_queue.cc:1048] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:27.859558 24363 rpcz_store.cc:276] 1023 20:15:25.639374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.639717 (+ 343us) service_pool.cc:225] Handling call
1023 20:15:27.622890 (+1983173us) ts_tablet_manager.cc:1151] Deleting tablet 5346a221c78342359eb6e418bd12cd24
1023 20:15:27.637476 (+ 14586us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:27.805643 (+168167us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.810399 (+ 4756us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:27.859461 (+ 49062us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1977658}
I20251023 20:15:28.053064 24349 tablet_replica.cc:333] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.053284 24349 raft_consensus.cc:2243] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.053563 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 5346a221c78342359eb6e418bd12cd24 (table default.Table_54bd013a97df4bb08fdab2eeea9f5d42 [id=71ba16e8b75642e09686eb46c4315366]) successfully deleted
I20251023 20:15:28.054965 24349 raft_consensus.cc:2272] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.056236 24349 ts_tablet_manager.cc:1916] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.063525 24184 consensus_queue.cc:1048] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.075734 23790 tablet_bootstrap.cc:492] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.075912 23790 ts_tablet_manager.cc:1403] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.242s user 0.010s sys 0.023s
I20251023 20:15:28.076678 23790 raft_consensus.cc:359] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.076866 23790 raft_consensus.cc:385] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.076920 23790 raft_consensus.cc:740] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:28.080456 23470 consensus_peers.cc:597] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8ba7ab28dfa4ac38e0b58ba2115495c. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.085282 23790 consensus_queue.cc:260] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.053678 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 79) took 2412 ms. Trace:
I20251023 20:15:28.054131 23787 ts_tablet_manager.cc:1434] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251023 20:15:28.087253 24349 ts_tablet_manager.cc:1929] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.087360 24349 log.cc:1199] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a21ec49194e24723b31fd9c86f1f85e4
I20251023 20:15:28.087440 23787 tablet_bootstrap.cc:492] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.087688 24349 ts_tablet_manager.cc:1950] T a21ec49194e24723b31fd9c86f1f85e4 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.089371 23790 ts_tablet_manager.cc:1434] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251023 20:15:28.089398 23787 tablet_bootstrap.cc:654] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.089643 23790 tablet_bootstrap.cc:492] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:28.092542 23333 consensus_peers.cc:597] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edec01a63a3046d9ae5a372fc9068b52. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.095530 24363 rpcz_store.cc:276] 1023 20:15:25.640730 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.641102 (+ 372us) service_pool.cc:225] Handling call
1023 20:15:28.053519 (+2412417us) ts_tablet_manager.cc:1151] Deleting tablet db758a7a7a89410f8ac1da72048fbc56
1023 20:15:28.053555 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.053655 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408256}
I20251023 20:15:28.096124 24363 tablet_replica.cc:333] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:28.096100 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
I20251023 20:15:28.096424 24363 raft_consensus.cc:2243] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.096805 24363 raft_consensus.cc:2272] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.098858 24363 ts_tablet_manager.cc:1916] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.102761 23790 tablet_bootstrap.cc:654] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.103567 24775 tablet.cc:1722] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:28.104521 24776 tablet.cc:1722] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:28.105638 23786 tablet_bootstrap.cc:492] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.105782 23786 ts_tablet_manager.cc:1403] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.207s user 0.014s sys 0.037s
I20251023 20:15:28.106499 23786 raft_consensus.cc:359] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.106673 23786 raft_consensus.cc:385] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.106734 23786 raft_consensus.cc:740] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:28.106966 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9a4f012469fa4e8e827e485287ad76dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a4f012469fa4e8e827e485287ad76dd already in progress: creating tablet
I20251023 20:15:28.107745 23786 consensus_queue.cc:260] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.109053 23786 ts_tablet_manager.cc:1434] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:28.109333 23786 tablet_bootstrap.cc:492] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.112054 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 120) took 1992 ms. Trace:
I20251023 20:15:28.113869 24665 consensus_queue.cc:1048] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.118934 24349 rpcz_store.cc:276] 1023 20:15:26.119737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.119849 (+ 112us) service_pool.cc:225] Handling call
1023 20:15:28.036123 (+1916274us) ts_tablet_manager.cc:1151] Deleting tablet a21ec49194e24723b31fd9c86f1f85e4
1023 20:15:28.053013 (+ 16890us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.060775 (+ 7762us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.087133 (+ 26358us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.112035 (+ 24902us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1915954}
I20251023 20:15:28.120553 24665 consensus_queue.cc:1048] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.132560 24655 raft_consensus.cc:493] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.132687 24655 raft_consensus.cc:515] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.133240 24655 leader_election.cc:290] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.133764 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46f700ea3db04a4bad539c943e850a44" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:28.134209 23605 leader_election.cc:343] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.128969 24785 tablet.cc:1722] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:28.117643 23786 tablet_bootstrap.cc:654] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.135507 23529 raft_consensus.cc:3060] T 37c8403b53f04a8e97ffe058e59b425c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.140408 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a21ec49194e24723b31fd9c86f1f85e4 (table default.Table_fc18b4fbf71a45f6b3cfee31effd8eef [id=ebb0bfc95f844d50859f36a09ac81f4d]) successfully deleted
I20251023 20:15:28.141471 24349 tablet_replica.cc:333] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.141676 24349 raft_consensus.cc:2243] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.141844 24349 raft_consensus.cc:2272] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.135957 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46f700ea3db04a4bad539c943e850a44" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.143018 24297 consensus_queue.cc:1048] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:28.143082 23604 leader_election.cc:343] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.143195 23604 leader_election.cc:304] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.143581 24659 raft_consensus.cc:2749] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.143754 24349 ts_tablet_manager.cc:1916] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.147315 24621 consensus_queue.cc:1048] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:28.154120 23333 consensus_peers.cc:597] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7645c36c983f438f8b1223256fd3ad2c. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.154743 24621 consensus_queue.cc:1048] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.156876 24423 webserver.cc:492] Webserver started at http://127.22.127.189:42879/ using document root <none> and password file <none>
I20251023 20:15:28.158214 24658 raft_consensus.cc:493] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.158440 24658 raft_consensus.cc:515] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.159389 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f908f086946949829cac15418a584730" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:28.159971 23605 leader_election.cc:343] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.160229 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f908f086946949829cac15418a584730" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:28.160660 23604 leader_election.cc:343] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.160774 23604 leader_election.cc:304] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.161008 24659 raft_consensus.cc:2749] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.161226 24658 leader_election.cc:290] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.161921 23787 tablet_bootstrap.cc:492] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.162047 23787 ts_tablet_manager.cc:1403] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.006s
I20251023 20:15:28.158100 24423 fs_manager.cc:362] Metadata directory not provided
I20251023 20:15:28.162741 23787 raft_consensus.cc:359] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.162931 23787 raft_consensus.cc:385] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.162989 23787 raft_consensus.cc:740] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.163193 23787 consensus_queue.cc:260] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.163702 23787 ts_tablet_manager.cc:1434] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:28.164014 23790 tablet_bootstrap.cc:492] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.164251 23787 tablet_bootstrap.cc:492] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.164901 24659 consensus_queue.cc:1048] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.166076 23790 ts_tablet_manager.cc:1403] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.000s
I20251023 20:15:28.173810 23786 tablet_bootstrap.cc:492] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.173959 23786 ts_tablet_manager.cc:1403] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.006s
I20251023 20:15:28.174683 23786 raft_consensus.cc:359] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.174857 23786 raft_consensus.cc:385] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.174918 23786 raft_consensus.cc:740] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.175135 23786 consensus_queue.cc:260] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.176343 23786 ts_tablet_manager.cc:1434] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:28.177208 23786 tablet_bootstrap.cc:492] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.178848 23786 tablet_bootstrap.cc:654] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.180344 23790 raft_consensus.cc:359] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.180521 23790 raft_consensus.cc:385] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.180572 23790 raft_consensus.cc:740] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.180745 23790 consensus_queue.cc:260] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.182315 23787 tablet_bootstrap.cc:654] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.182488 23790 ts_tablet_manager.cc:1434] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:28.182793 23790 tablet_bootstrap.cc:492] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.182826 24363 ts_tablet_manager.cc:1929] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.182931 24363 log.cc:1199] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/1e8eb7603c3e46f1b15cdff7a7bbedcf
I20251023 20:15:28.183547 24363 ts_tablet_manager.cc:1950] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.184319 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 80) took 2542 ms. Trace:
I20251023 20:15:28.184454 23790 tablet_bootstrap.cc:654] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.184448 24363 rpcz_store.cc:276] 1023 20:15:25.641345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.641598 (+ 253us) service_pool.cc:225] Handling call
1023 20:15:28.096063 (+2454465us) ts_tablet_manager.cc:1151] Deleting tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf
1023 20:15:28.096080 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.167984 (+ 71904us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.182385 (+ 14401us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.184299 (+ 1914us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2450767}
I20251023 20:15:28.185007 24363 tablet_replica.cc:333] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.184978 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf (table default.Table_1fd972f3964245fab674e2cfc26a8210 [id=cf5e426caddf4508bc0780b9809fff5e]) successfully deleted
I20251023 20:15:28.185267 24363 raft_consensus.cc:2243] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.185556 24259 raft_consensus.cc:493] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.185698 24259 raft_consensus.cc:515] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.186287 24259 leader_election.cc:290] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:28.186975 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.187199 23670 raft_consensus.cc:2393] T d6b0a4db18df490a966e8c475ddd14f4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:28.189164 24349 ts_tablet_manager.cc:1929] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.189267 24349 log.cc:1199] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/40e0301fb5974ec5b6e8291230a736c2
I20251023 20:15:28.189510 24349 ts_tablet_manager.cc:1950] T 40e0301fb5974ec5b6e8291230a736c2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.190172 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 121) took 2069 ms. Trace:
I20251023 20:15:28.190263 24349 rpcz_store.cc:276] 1023 20:15:26.120511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.120908 (+ 397us) service_pool.cc:225] Handling call
1023 20:15:28.141252 (+2020344us) ts_tablet_manager.cc:1151] Deleting tablet 40e0301fb5974ec5b6e8291230a736c2
1023 20:15:28.141270 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.157582 (+ 16312us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.189039 (+ 31457us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.190154 (+ 1115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019600}
I20251023 20:15:28.190696 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 40e0301fb5974ec5b6e8291230a736c2 (table default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]) successfully deleted
I20251023 20:15:28.185452 24363 raft_consensus.cc:2272] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.191365 24363 ts_tablet_manager.cc:1916] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.192692 23786 tablet_bootstrap.cc:492] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
W20251023 20:15:28.193670 23333 consensus_peers.cc:597] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.193714 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6b0a4db18df490a966e8c475ddd14f4" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.194617 23471 leader_election.cc:304] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:28.194972 24259 raft_consensus.cc:3060] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.195931 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:28.196148 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:28.196352 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:28.196703 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:28.196993 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:28.197358 23786 ts_tablet_manager.cc:1403] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
W20251023 20:15:28.197384 23333 consensus_peers.cc:597] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5964da70989b4e6cb77fe5d0a52ce9cf. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.197425 24259 raft_consensus.cc:2749] T d6b0a4db18df490a966e8c475ddd14f4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:28.197933 23333 consensus_peers.cc:597] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.198092 23786 raft_consensus.cc:359] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.198248 23786 raft_consensus.cc:385] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.198304 23786 raft_consensus.cc:740] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.198518 23786 consensus_queue.cc:260] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.200940 23787 tablet_bootstrap.cc:492] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.201112 23787 ts_tablet_manager.cc:1403] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20251023 20:15:28.202194 23787 raft_consensus.cc:359] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.202349 23787 raft_consensus.cc:385] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.202396 23787 raft_consensus.cc:740] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.202464 23786 ts_tablet_manager.cc:1434] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:28.202617 23787 consensus_queue.cc:260] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.202971 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 122) took 2081 ms. Trace:
I20251023 20:15:28.203058 24349 rpcz_store.cc:276] 1023 20:15:26.121268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.122447 (+ 1179us) service_pool.cc:225] Handling call
1023 20:15:28.202855 (+2080408us) ts_tablet_manager.cc:1151] Deleting tablet 25079532547d4175802b84877790662d
1023 20:15:28.202863 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.202954 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2080048}
I20251023 20:15:28.204778 23786 tablet_bootstrap.cc:492] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.205755 23787 ts_tablet_manager.cc:1434] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20251023 20:15:28.205826 23333 consensus_peers.cc:597] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.206553 23787 tablet_bootstrap.cc:492] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:28.207324 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 25079532547d4175802b84877790662d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25079532547d4175802b84877790662d already in progress: creating tablet
I20251023 20:15:28.207887 24349 tablet_replica.cc:333] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.208071 24349 raft_consensus.cc:2243] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.208233 24349 raft_consensus.cc:2272] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.208348 23787 tablet_bootstrap.cc:654] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.208248 24363 ts_tablet_manager.cc:1929] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.208568 24363 log.cc:1199] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a21ec49194e24723b31fd9c86f1f85e4
I20251023 20:15:28.208714 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:28.209106 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:28.211107 24349 ts_tablet_manager.cc:1916] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.211675 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:28.213316 24363 ts_tablet_manager.cc:1950] T a21ec49194e24723b31fd9c86f1f85e4 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.213732 23790 tablet_bootstrap.cc:492] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.213874 23790 ts_tablet_manager.cc:1403] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20251023 20:15:28.214020 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 81) took 2572 ms. Trace:
I20251023 20:15:28.214036 24423 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251023 20:15:28.214107 24363 rpcz_store.cc:276] 1023 20:15:25.642056 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.644440 (+ 2384us) service_pool.cc:225] Handling call
1023 20:15:28.184958 (+2540518us) ts_tablet_manager.cc:1151] Deleting tablet a21ec49194e24723b31fd9c86f1f85e4
1023 20:15:28.184968 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.201639 (+ 16671us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.207872 (+ 6233us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.214002 (+ 6130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2539640}
I20251023 20:15:28.214486 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 70) took 2579 ms. Trace:
I20251023 20:15:28.214567 24363 rpcz_store.cc:276] 1023 20:15:25.635367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.636278 (+ 911us) service_pool.cc:225] Handling call
1023 20:15:28.214364 (+2578086us) ts_tablet_manager.cc:1151] Deleting tablet 1822fdf1f5f44dd4a742aa3c2f8443cf
1023 20:15:28.214374 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.214469 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2569027}
I20251023 20:15:28.214847 24363 tablet_replica.cc:333] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.214630 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a21ec49194e24723b31fd9c86f1f85e4 (table default.Table_fc18b4fbf71a45f6b3cfee31effd8eef [id=ebb0bfc95f844d50859f36a09ac81f4d]) successfully deleted
I20251023 20:15:28.215051 24363 raft_consensus.cc:2243] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:28.215235 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
I20251023 20:15:28.215348 24363 raft_consensus.cc:2272] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.214251 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:28.215736 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:28.215852 24363 ts_tablet_manager.cc:1916] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.216005 23786 tablet_bootstrap.cc:654] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.216101 23790 raft_consensus.cc:359] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.216274 23790 raft_consensus.cc:385] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.216332 23790 raft_consensus.cc:740] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.216558 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:28.217689 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:28.218151 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:28.218616 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:28.219734 23790 consensus_queue.cc:260] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.220976 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:28.221599 23790 ts_tablet_manager.cc:1434] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20251023 20:15:28.221884 23790 tablet_bootstrap.cc:492] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.224289 23790 tablet_bootstrap.cc:654] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.225363 23786 tablet_bootstrap.cc:492] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.226032 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:28.226300 23786 ts_tablet_manager.cc:1403] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251023 20:15:28.232019 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:28.232645 24363 ts_tablet_manager.cc:1929] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.232815 24363 log.cc:1199] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/40e0301fb5974ec5b6e8291230a736c2
I20251023 20:15:28.232915 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:28.233335 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:28.227138 23786 raft_consensus.cc:359] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.237540 24363 ts_tablet_manager.cc:1950] T 40e0301fb5974ec5b6e8291230a736c2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.238399 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 82) took 2593 ms. Trace:
I20251023 20:15:28.238545 24363 rpcz_store.cc:276] 1023 20:15:25.644774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.649330 (+ 4556us) service_pool.cc:225] Handling call
1023 20:15:28.214810 (+2565480us) ts_tablet_manager.cc:1151] Deleting tablet 40e0301fb5974ec5b6e8291230a736c2
1023 20:15:28.214818 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.218683 (+ 3865us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.232525 (+ 13842us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.238374 (+ 5849us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2565233}
I20251023 20:15:28.239197 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 83) took 2589 ms. Trace:
I20251023 20:15:28.239332 24363 rpcz_store.cc:276] 1023 20:15:25.649820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.650172 (+ 352us) service_pool.cc:225] Handling call
1023 20:15:28.239063 (+2588891us) ts_tablet_manager.cc:1151] Deleting tablet 25079532547d4175802b84877790662d
1023 20:15:28.239074 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.239180 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2588725}
I20251023 20:15:28.239782 24363 tablet_replica.cc:333] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.240002 24363 raft_consensus.cc:2243] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.240198 24363 raft_consensus.cc:2272] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.241714 23786 raft_consensus.cc:385] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.241804 23786 raft_consensus.cc:740] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.243135 24363 ts_tablet_manager.cc:1916] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.239037 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 40e0301fb5974ec5b6e8291230a736c2 (table default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]) successfully deleted
W20251023 20:15:28.243460 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 25079532547d4175802b84877790662d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25079532547d4175802b84877790662d already in progress: creating tablet
I20251023 20:15:28.243888 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:28.244104 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:28.244395 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:28.244633 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:28.244882 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:28.245028 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:28.245327 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:28.245491 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:28.245635 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:28.246030 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:28.246168 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:28.246579 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:28.246877 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:28.247045 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:28.247596 23786 consensus_queue.cc:260] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.249253 23786 ts_tablet_manager.cc:1434] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251023 20:15:28.250911 23790 tablet_bootstrap.cc:492] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.251070 23790 ts_tablet_manager.cc:1403] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20251023 20:15:28.251881 23790 raft_consensus.cc:359] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.252056 23790 raft_consensus.cc:385] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.252115 23790 raft_consensus.cc:740] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.253727 24363 ts_tablet_manager.cc:1929] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.253839 24363 log.cc:1199] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/ac55e14745534a02a2f6a422cab6b149
I20251023 20:15:28.254752 24363 ts_tablet_manager.cc:1950] T ac55e14745534a02a2f6a422cab6b149 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.255193 23790 consensus_queue.cc:260] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.256098 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 84) took 2605 ms. Trace:
I20251023 20:15:28.257732 23790 ts_tablet_manager.cc:1434] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:28.258107 24363 rpcz_store.cc:276] 1023 20:15:25.650561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.652183 (+ 1622us) service_pool.cc:225] Handling call
1023 20:15:28.239727 (+2587544us) ts_tablet_manager.cc:1151] Deleting tablet ac55e14745534a02a2f6a422cab6b149
1023 20:15:28.239736 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.246171 (+ 6435us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.253621 (+ 7450us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.256080 (+ 2459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2586147}
I20251023 20:15:28.259192 23790 tablet_bootstrap.cc:492] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.259441 24363 tablet_replica.cc:333] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.259549 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet ac55e14745534a02a2f6a422cab6b149 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]) successfully deleted
I20251023 20:15:28.261263 24363 raft_consensus.cc:2243] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.261675 23786 tablet_bootstrap.cc:492] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.261677 24363 raft_consensus.cc:2272] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.263303 23786 tablet_bootstrap.cc:654] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.265877 24363 ts_tablet_manager.cc:1916] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.267794 23787 tablet_bootstrap.cc:492] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.267980 23787 ts_tablet_manager.cc:1403] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.008s
I20251023 20:15:28.268566 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:28.270025 23790 tablet_bootstrap.cc:654] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.270619 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:28.270776 23787 raft_consensus.cc:359] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.270941 23787 raft_consensus.cc:385] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.270995 23787 raft_consensus.cc:740] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.271212 23787 consensus_queue.cc:260] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.271678 23787 ts_tablet_manager.cc:1434] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:28.271950 23787 tablet_bootstrap.cc:492] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.274667 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:28.275599 23787 tablet_bootstrap.cc:654] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.276355 24363 ts_tablet_manager.cc:1929] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.276453 24363 log.cc:1199] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/fe922c438a894a7aa04699b2d2897a28
I20251023 20:15:28.276750 24363 ts_tablet_manager.cc:1950] T fe922c438a894a7aa04699b2d2897a28 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.277073 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:28.281318 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 85) took 2628 ms. Trace:
I20251023 20:15:28.282299 24363 rpcz_store.cc:276] 1023 20:15:25.653186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.653370 (+ 184us) service_pool.cc:225] Handling call
1023 20:15:28.259395 (+2606025us) ts_tablet_manager.cc:1151] Deleting tablet fe922c438a894a7aa04699b2d2897a28
1023 20:15:28.259405 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.272575 (+ 13170us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.276255 (+ 3680us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.281292 (+ 5037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2605593}
I20251023 20:15:28.283000 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:28.283041 24621 raft_consensus.cc:493] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.283111 23790 tablet_bootstrap.cc:492] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.283227 23790 ts_tablet_manager.cc:1403] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.001s
I20251023 20:15:28.283160 24621 raft_consensus.cc:515] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.283732 24621 leader_election.cc:290] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.284112 24363 tablet_replica.cc:333] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.284256 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet fe922c438a894a7aa04699b2d2897a28 (table default.Table_aab23aa8d50a4902b09495b3fa1ea6ca [id=fe2df9a958ec4863ab2100750ba7dc5e]) successfully deleted
I20251023 20:15:28.284353 24363 raft_consensus.cc:2243] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.284511 24363 raft_consensus.cc:2272] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.285071 23786 tablet_bootstrap.cc:492] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.285226 23786 ts_tablet_manager.cc:1403] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251023 20:15:28.285802 24363 ts_tablet_manager.cc:1916] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.285902 23786 raft_consensus.cc:359] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.286062 23786 raft_consensus.cc:385] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.286108 23786 raft_consensus.cc:740] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.286294 23786 consensus_queue.cc:260] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:28.286510 23333 leader_election.cc:343] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:28.286566 23790 raft_consensus.cc:359] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.287422 23786 ts_tablet_manager.cc:1434] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251023 20:15:28.286723 23333 leader_election.cc:343] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:28.287587 23333 leader_election.cc:304] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.288008 24621 raft_consensus.cc:2749] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:28.288105 23786 tablet_bootstrap.cc:492] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.286392 24423 fs_manager.cc:714] Time spent opening directory manager: real 0.062s user 0.001s sys 0.005s
I20251023 20:15:28.290187 23787 tablet_bootstrap.cc:492] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.290314 23787 ts_tablet_manager.cc:1403] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20251023 20:15:28.291004 23787 raft_consensus.cc:359] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.291155 23787 raft_consensus.cc:385] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.291206 23787 raft_consensus.cc:740] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.292860 23786 tablet_bootstrap.cc:654] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.292994 24363 ts_tablet_manager.cc:1929] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.293097 24363 log.cc:1199] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a27f192792cf41288d69ca15ab538472
I20251023 20:15:28.293411 24363 ts_tablet_manager.cc:1950] T a27f192792cf41288d69ca15ab538472 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.293965 23787 consensus_queue.cc:260] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.287607 23790 raft_consensus.cc:385] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.294220 23790 raft_consensus.cc:740] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.294430 23790 consensus_queue.cc:260] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.294559 23787 ts_tablet_manager.cc:1434] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:28.295174 23790 ts_tablet_manager.cc:1434] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251023 20:15:28.294813 23787 tablet_bootstrap.cc:492] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.294055 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 86) took 2640 ms. Trace:
I20251023 20:15:28.295806 24363 rpcz_store.cc:276] 1023 20:15:25.653332 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.654094 (+ 762us) service_pool.cc:225] Handling call
1023 20:15:28.284067 (+2629973us) ts_tablet_manager.cc:1151] Deleting tablet a27f192792cf41288d69ca15ab538472
1023 20:15:28.284075 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.290213 (+ 6138us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.292904 (+ 2691us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.294041 (+ 1137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2629107}
I20251023 20:15:28.296514 24363 tablet_replica.cc:333] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.296523 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a27f192792cf41288d69ca15ab538472 (table default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]) successfully deleted
I20251023 20:15:28.296691 24363 raft_consensus.cc:2243] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.296859 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:28.296948 23790 tablet_bootstrap.cc:492] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.297055 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:28.297554 24363 raft_consensus.cc:2272] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.297585 23787 tablet_bootstrap.cc:654] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.297978 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:28.298385 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:28.298362 24363 ts_tablet_manager.cc:1916] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.299342 23218 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:28.300551 23213 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet d62dbf044acd4459b26d2e7e50f6206e
I20251023 20:15:28.301116 23213 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 49) took 3045 ms. Trace:
I20251023 20:15:28.301225 23213 rpcz_store.cc:276] 1023 20:15:25.255246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.476009 (+1220763us) service_pool.cc:225] Handling call
1023 20:15:26.538336 (+ 62327us) spinlock_profiling.cc:243] Waited 62.1 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0603796 00007f8997108aa6 00007f89971087d5 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:26.538801 (+ 465us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:26.538953 (+ 152us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:26.539031 (+ 78us) catalog_manager.cc:2824] Locking tablets
1023 20:15:26.553710 (+ 14679us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:26.561639 (+ 7929us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:26.561708 (+ 69us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:28.300504 (+1738796us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:28.301041 (+ 537us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.554461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.554585 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.554591 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:26.554594 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:26.554714 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.554727 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.554732 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.554834 (+ 102us) tablet.cc:699] Row locks acquired
1023 20:15:26.554837 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:26.554871 (+ 34us) write_op.cc:270] Start()
1023 20:15:26.554902 (+ 31us) write_op.cc:276] Timestamp: P: 1761250526554851 usec, L: 0
1023 20:15:26.554906 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.555079 (+ 173us) log.cc:844] Serialized 734 byte log entry
1023 20:15:26.561000 (+ 5921us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.561153 (+ 153us) write_op.cc:301] APPLY: starting
1023 20:15:26.561167 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.561208 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.561230 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.561297 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.561334 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:26.561344 (+ 10us) write_op.cc:312] APPLY: finished
1023 20:15:26.561409 (+ 65us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.561464 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.561480 (+ 16us) write_op.cc:454] Released schema lock
1023 20:15:26.561486 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.561546 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.561549 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":142735232,"child_traces":[["op",{"apply.queue_time_us":104,"apply.run_cpu_time_us":387,"apply.run_wall_time_us":386,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":736,"raft.queue_time_us":191,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":237,"replication_time_us":6065}]]}
I20251023 20:15:28.302485 23790 tablet_bootstrap.cc:654] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.305951 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.306455 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.306695 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:28.307081 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d62dbf044acd4459b26d2e7e50f6206e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d62dbf044acd4459b26d2e7e50f6206e already in progress: creating tablet
W20251023 20:15:28.312937 23333 consensus_peers.cc:597] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4c8dcbdfcf4a15b8a4194ea04b93a0. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.313697 23786 tablet_bootstrap.cc:492] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.313846 23786 ts_tablet_manager.cc:1403] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20251023 20:15:28.313910 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973] alter complete (version 2)
I20251023 20:15:28.314503 23786 raft_consensus.cc:359] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.314661 23786 raft_consensus.cc:385] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.314703 23786 raft_consensus.cc:740] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.314896 23786 consensus_queue.cc:260] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.315346 23786 ts_tablet_manager.cc:1434] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:28.315738 23786 tablet_bootstrap.cc:492] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.316970 23786 tablet_bootstrap.cc:654] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.318742 24363 ts_tablet_manager.cc:1929] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.318856 24363 log.cc:1199] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a708cd011802458aa0e2991fd4b4a2d3
I20251023 20:15:28.319160 24363 ts_tablet_manager.cc:1950] T a708cd011802458aa0e2991fd4b4a2d3 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.319837 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 89) took 2665 ms. Trace:
I20251023 20:15:28.319923 24363 rpcz_store.cc:276] 1023 20:15:25.654578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.654779 (+ 201us) service_pool.cc:225] Handling call
1023 20:15:28.296473 (+2641694us) ts_tablet_manager.cc:1151] Deleting tablet a708cd011802458aa0e2991fd4b4a2d3
1023 20:15:28.296482 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.307438 (+ 10956us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.318625 (+ 11187us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.319816 (+ 1191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2641350}
I20251023 20:15:28.320801 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a708cd011802458aa0e2991fd4b4a2d3 (table default.Table_d2b87a7247624b0380ee21ec002b9f77 [id=f64a3298a8d44a7ba1700cac9bf91080]) successfully deleted
I20251023 20:15:28.321933 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477] alter complete (version 2)
W20251023 20:15:28.324095 23211 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd548fd9de284377ab557bd98f621068 consensus state has no leader
I20251023 20:15:28.326756 24349 ts_tablet_manager.cc:1929] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.326860 24349 log.cc:1199] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/fe922c438a894a7aa04699b2d2897a28
I20251023 20:15:28.327154 24349 ts_tablet_manager.cc:1950] T fe922c438a894a7aa04699b2d2897a28 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.327829 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 123) took 2204 ms. Trace:
I20251023 20:15:28.327903 24349 rpcz_store.cc:276] 1023 20:15:26.122939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.124476 (+ 1537us) service_pool.cc:225] Handling call
1023 20:15:28.207842 (+2083366us) ts_tablet_manager.cc:1151] Deleting tablet fe922c438a894a7aa04699b2d2897a28
1023 20:15:28.207851 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.323722 (+115871us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.326667 (+ 2945us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.327810 (+ 1143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2083134}
I20251023 20:15:28.328187 24349 tablet_replica.cc:333] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.328353 24349 raft_consensus.cc:2243] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.328485 24349 raft_consensus.cc:2272] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.328986 24349 ts_tablet_manager.cc:1916] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.329262 24363 tablet_replica.cc:333] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.329444 24363 raft_consensus.cc:2243] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.329583 24363 raft_consensus.cc:2272] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.338241 24363 ts_tablet_manager.cc:1916] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.340588 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet fe922c438a894a7aa04699b2d2897a28 (table default.Table_aab23aa8d50a4902b09495b3fa1ea6ca [id=fe2df9a958ec4863ab2100750ba7dc5e]) successfully deleted
I20251023 20:15:28.346566 24363 ts_tablet_manager.cc:1929] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.346693 24363 log.cc:1199] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/15ced74d0dd74e5195ed98358957d0dd
W20251023 20:15:28.348738 23333 consensus_peers.cc:597] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5964da70989b4e6cb77fe5d0a52ce9cf. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.349052 23215 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582] alter complete (version 1)
I20251023 20:15:28.349234 24363 ts_tablet_manager.cc:1950] T 15ced74d0dd74e5195ed98358957d0dd P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.351197 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 88) took 2696 ms. Trace:
I20251023 20:15:28.351282 24363 rpcz_store.cc:276] 1023 20:15:25.654483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.654617 (+ 134us) service_pool.cc:225] Handling call
1023 20:15:28.329202 (+2674585us) ts_tablet_manager.cc:1151] Deleting tablet 15ced74d0dd74e5195ed98358957d0dd
1023 20:15:28.329214 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.341301 (+ 12087us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.346466 (+ 5165us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.351176 (+ 4710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2674015}
I20251023 20:15:28.352380 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 15ced74d0dd74e5195ed98358957d0dd (table default.Table_e62c8689605f4db7b6e5016d5dcda175 [id=91e8d69e8f264c6dafd1173e05f7be06]) successfully deleted
I20251023 20:15:28.352751 24658 raft_consensus.cc:493] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.352886 24658 raft_consensus.cc:515] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.353547 24658 leader_election.cc:290] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.354338 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83dae34129794f49b43038d3b1636ea0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.354429 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83dae34129794f49b43038d3b1636ea0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:28.354936 23604 leader_election.cc:343] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:28.355278 23605 leader_election.cc:343] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.355360 23605 leader_election.cc:304] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.355919 24658 raft_consensus.cc:2749] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.356307 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 90) took 2695 ms. Trace:
I20251023 20:15:28.357118 24363 rpcz_store.cc:276] 1023 20:15:25.661222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.661270 (+ 48us) service_pool.cc:225] Handling call
1023 20:15:28.356186 (+2694916us) ts_tablet_manager.cc:1151] Deleting tablet fbab01ecb27a426188689408079eaa01
1023 20:15:28.356197 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.356293 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2694686}
I20251023 20:15:28.357676 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 91) took 2696 ms. Trace:
I20251023 20:15:28.357810 24363 rpcz_store.cc:276] 1023 20:15:25.661676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.661830 (+ 154us) service_pool.cc:225] Handling call
1023 20:15:28.357573 (+2695743us) ts_tablet_manager.cc:1151] Deleting tablet ea0cbd6bec674cf5831486f500cc7b36
1023 20:15:28.357582 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.357662 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2695490}
I20251023 20:15:28.360077 24349 ts_tablet_manager.cc:1929] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.360466 24363 tablet_replica.cc:333] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:28.361373 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
W20251023 20:15:28.362787 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
I20251023 20:15:28.363049 23215 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6] alter complete (version 1)
I20251023 20:15:28.365270 24363 raft_consensus.cc:2243] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.365453 24363 raft_consensus.cc:2272] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.366000 24363 ts_tablet_manager.cc:1916] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.367494 24349 log.cc:1199] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/585a9af140d9479fb83d13e68a62300c
I20251023 20:15:28.367550 23210 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 377de4ea0bd6489284fac6b0fab52c8c
I20251023 20:15:28.367839 24349 ts_tablet_manager.cc:1950] T 585a9af140d9479fb83d13e68a62300c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.368590 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 124) took 2243 ms. Trace:
I20251023 20:15:28.368681 24349 rpcz_store.cc:276] 1023 20:15:26.124833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.125315 (+ 482us) service_pool.cc:225] Handling call
1023 20:15:28.328143 (+2202828us) ts_tablet_manager.cc:1151] Deleting tablet 585a9af140d9479fb83d13e68a62300c
1023 20:15:28.328152 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.347912 (+ 19760us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.359960 (+ 12048us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.368568 (+ 8608us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2202176}
I20251023 20:15:28.368782 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.368767 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.368990 24349 tablet_replica.cc:333] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.369217 24349 raft_consensus.cc:2243] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.369381 24349 raft_consensus.cc:2272] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.369541 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.370769 24495 consensus_queue.cc:1048] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:28.371511 23333 consensus_peers.cc:597] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fbad98d3ab24cc5b7726a92fba04cf8. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.373162 24259 consensus_queue.cc:1048] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.373288 24349 ts_tablet_manager.cc:1916] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.376168 23210 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 55) took 2799 ms. Trace:
W20251023 20:15:28.376600 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 377de4ea0bd6489284fac6b0fab52c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377de4ea0bd6489284fac6b0fab52c8c already in progress: creating tablet
I20251023 20:15:28.376294 23210 rpcz_store.cc:276] 1023 20:15:25.568424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.576866 (+1008442us) service_pool.cc:225] Handling call
1023 20:15:26.577865 (+ 999us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:26.578007 (+ 142us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:26.578070 (+ 63us) catalog_manager.cc:2824] Locking tablets
1023 20:15:27.013285 (+435215us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:27.036542 (+ 23257us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:27.036635 (+ 93us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:28.367342 (+1330707us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:28.368056 (+ 714us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:27.014568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:27.014707 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:27.014714 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:27.014717 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:27.014864 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:27.014881 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:27.014886 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:27.014996 (+ 110us) tablet.cc:699] Row locks acquired
1023 20:15:27.014998 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:27.015033 (+ 35us) write_op.cc:270] Start()
1023 20:15:27.015068 (+ 35us) write_op.cc:276] Timestamp: P: 1761250527014981 usec, L: 0
1023 20:15:27.015071 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:27.015250 (+ 179us) log.cc:844] Serialized 765 byte log entry
1023 20:15:27.035292 (+ 20042us) op_driver.cc:464] REPLICATION: finished
1023 20:15:27.035435 (+ 143us) write_op.cc:301] APPLY: starting
1023 20:15:27.035451 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:27.035506 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:27.035545 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:27.035629 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:27.035669 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:27.035678 (+ 9us) write_op.cc:312] APPLY: finished
1023 20:15:27.035778 (+ 100us) log.cc:844] Serialized 29 byte log entry
1023 20:15:27.035865 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:27.035890 (+ 25us) write_op.cc:454] Released schema lock
1023 20:15:27.035898 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:27.035970 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:27.035975 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":525,"apply.run_wall_time_us":525,"num_ops":2,"prepare.queue_time_us":376,"prepare.run_cpu_time_us":785,"prepare.run_wall_time_us":785,"raft.queue_time_us":138,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":257,"replication_time_us":20186}]]}
I20251023 20:15:28.381189 24658 raft_consensus.cc:493] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.381338 24658 raft_consensus.cc:515] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.381947 24658 leader_election.cc:290] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.382112 24670 consensus_queue.cc:1048] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.382712 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8eaa2167f82475ca2f6a110aba288ec" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.382722 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8eaa2167f82475ca2f6a110aba288ec" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:28.383168 23604 leader_election.cc:343] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:28.383288 23605 leader_election.cc:343] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.383378 23605 leader_election.cc:304] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.383849 24655 raft_consensus.cc:2749] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.385131 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 585a9af140d9479fb83d13e68a62300c (table default.Table_c3d2d37dd2bd414fae4c968fc3f52a25 [id=97396cb497da42f6af247dd68249182e]) successfully deleted
I20251023 20:15:28.398187 24363 ts_tablet_manager.cc:1929] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.398401 24363 log.cc:1199] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/2532c8567db14901b383f42e5f04b4c2
I20251023 20:15:28.398840 24363 ts_tablet_manager.cc:1950] T 2532c8567db14901b383f42e5f04b4c2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.398950 24670 consensus_queue.cc:1048] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.399784 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 92) took 2738 ms. Trace:
I20251023 20:15:28.400003 24363 rpcz_store.cc:276] 1023 20:15:25.661793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.695073 (+ 33280us) service_pool.cc:225] Handling call
1023 20:15:28.360417 (+2665344us) ts_tablet_manager.cc:1151] Deleting tablet 2532c8567db14901b383f42e5f04b4c2
1023 20:15:28.360427 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.371789 (+ 11362us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.395077 (+ 23288us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.399752 (+ 4675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2665105}
I20251023 20:15:28.400645 24363 tablet_replica.cc:333] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.400911 24363 raft_consensus.cc:2243] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.401172 24363 raft_consensus.cc:2272] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.402084 24363 ts_tablet_manager.cc:1916] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.402379 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 2532c8567db14901b383f42e5f04b4c2 (table default.Table_bf99025eb142454fa0513fafcbe636aa [id=f52f1d83bd9345ffa08e6d3d282914e0]) successfully deleted
W20251023 20:15:28.403407 23333 consensus_peers.cc:597] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe922c438a894a7aa04699b2d2897a28. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.405231 24349 ts_tablet_manager.cc:1929] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.405411 24349 log.cc:1199] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a27f192792cf41288d69ca15ab538472
I20251023 20:15:28.405807 24349 ts_tablet_manager.cc:1950] T a27f192792cf41288d69ca15ab538472 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.406582 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 125) took 2280 ms. Trace:
I20251023 20:15:28.408578 24349 rpcz_store.cc:276] 1023 20:15:26.126104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.126274 (+ 170us) service_pool.cc:225] Handling call
1023 20:15:28.368943 (+2242669us) ts_tablet_manager.cc:1151] Deleting tablet a27f192792cf41288d69ca15ab538472
1023 20:15:28.368952 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.395643 (+ 26691us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.403477 (+ 7834us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.406558 (+ 3081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2241990}
I20251023 20:15:28.409296 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a27f192792cf41288d69ca15ab538472 (table default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]) successfully deleted
I20251023 20:15:28.409775 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 126) took 2282 ms. Trace:
I20251023 20:15:28.412070 24349 rpcz_store.cc:276] 1023 20:15:26.127095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.127593 (+ 498us) service_pool.cc:225] Handling call
1023 20:15:28.409626 (+2282033us) ts_tablet_manager.cc:1151] Deleting tablet b90810781f724c78935a33d868ffa27d
1023 20:15:28.409638 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.409746 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2281764}
W20251023 20:15:28.412708 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
I20251023 20:15:28.413121 24349 tablet_replica.cc:333] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.413358 24349 raft_consensus.cc:2243] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.413560 24349 raft_consensus.cc:2272] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.414166 24349 ts_tablet_manager.cc:1916] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.416851 24670 raft_consensus.cc:493] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.417009 24670 raft_consensus.cc:515] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.417729 24670 leader_election.cc:290] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.418658 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af899c01b4af46d39d27f984cc3d5940" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:28.419366 23333 leader_election.cc:343] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.419687 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af899c01b4af46d39d27f984cc3d5940" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.419893 23668 raft_consensus.cc:2468] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.420460 23532 raft_consensus.cc:3060] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.421268 23333 leader_election.cc:304] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.421794 24670 raft_consensus.cc:2804] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.421870 24670 raft_consensus.cc:493] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.421922 24670 raft_consensus.cc:3060] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.424446 24670 raft_consensus.cc:515] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.425253 24349 ts_tablet_manager.cc:1929] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.425354 24349 log.cc:1199] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/283166bbd8004ce3bca9f2ecbdbcbf48
I20251023 20:15:28.425662 24349 ts_tablet_manager.cc:1950] T 283166bbd8004ce3bca9f2ecbdbcbf48 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.425855 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af899c01b4af46d39d27f984cc3d5940" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.426092 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af899c01b4af46d39d27f984cc3d5940" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:28.426273 23668 raft_consensus.cc:3060] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.426378 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 127) took 2298 ms. Trace:
I20251023 20:15:28.426446 24349 rpcz_store.cc:276] 1023 20:15:26.128071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.129757 (+ 1686us) service_pool.cc:225] Handling call
1023 20:15:28.413046 (+2283289us) ts_tablet_manager.cc:1151] Deleting tablet 283166bbd8004ce3bca9f2ecbdbcbf48
1023 20:15:28.413057 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.422484 (+ 9427us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.425156 (+ 2672us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.426361 (+ 1205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2283069}
I20251023 20:15:28.426865 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 283166bbd8004ce3bca9f2ecbdbcbf48 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:28.427914 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c] alter complete (version 3)
I20251023 20:15:28.428104 24363 ts_tablet_manager.cc:1929] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.428193 24363 log.cc:1199] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d951134f6773410386bc57f7f90310c3
I20251023 20:15:28.428467 24363 ts_tablet_manager.cc:1950] T d951134f6773410386bc57f7f90310c3 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.428443 23215 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d] alter complete (version 4)
I20251023 20:15:28.428684 23668 raft_consensus.cc:2468] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:28.433439 23333 leader_election.cc:343] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.433887 23333 leader_election.cc:304] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.434275 24665 consensus_queue.cc:1048] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.434672 24665 raft_consensus.cc:2804] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.434769 24665 raft_consensus.cc:697] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.434888 24259 raft_consensus.cc:493] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.435003 24259 raft_consensus.cc:515] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.434990 24665 consensus_queue.cc:237] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.435546 24259 leader_election.cc:290] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:28.436800 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 93) took 2741 ms. Trace:
I20251023 20:15:28.436895 24363 rpcz_store.cc:276] 1023 20:15:25.695558 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.695603 (+ 45us) service_pool.cc:225] Handling call
1023 20:15:28.400597 (+2704994us) ts_tablet_manager.cc:1151] Deleting tablet d951134f6773410386bc57f7f90310c3
1023 20:15:28.400607 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.408301 (+ 7694us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.428001 (+ 19700us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.436776 (+ 8775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704808}
I20251023 20:15:28.437450 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d951134f6773410386bc57f7f90310c3 (table default.Table_deb8b605c159407ab25c527a038c8d44 [id=7313119e56d94c05b9d8a31709af6ff6]) successfully deleted
I20251023 20:15:28.437996 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 87) took 2783 ms. Trace:
I20251023 20:15:28.440601 24363 rpcz_store.cc:276] 1023 20:15:25.654354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.706737 (+ 52383us) service_pool.cc:225] Handling call
1023 20:15:28.437861 (+2731124us) ts_tablet_manager.cc:1151] Deleting tablet b90810781f724c78935a33d868ffa27d
1023 20:15:28.437872 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.437978 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2730524}
W20251023 20:15:28.441308 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
I20251023 20:15:28.441720 24363 tablet_replica.cc:333] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.425045 24670 leader_election.cc:290] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.441962 24363 raft_consensus.cc:2243] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.442171 24363 raft_consensus.cc:2272] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.442782 24363 ts_tablet_manager.cc:1916] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.448248 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.438349 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.448376 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.448475 23665 raft_consensus.cc:2393] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:28.449566 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.449759 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.449949 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.450102 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.450320 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.450525 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.450790 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.451218 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.455319 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 128) took 2325 ms. Trace:
I20251023 20:15:28.457122 24349 rpcz_store.cc:276] 1023 20:15:26.130094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.131905 (+ 1811us) service_pool.cc:225] Handling call
1023 20:15:28.455174 (+2323269us) ts_tablet_manager.cc:1151] Deleting tablet fbab01ecb27a426188689408079eaa01
1023 20:15:28.455187 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.455296 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2323010}
W20251023 20:15:28.457736 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
I20251023 20:15:28.458233 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 129) took 2325 ms. Trace:
I20251023 20:15:28.458354 24349 rpcz_store.cc:276] 1023 20:15:26.132322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.132770 (+ 448us) service_pool.cc:225] Handling call
1023 20:15:28.458106 (+2325336us) ts_tablet_manager.cc:1151] Deleting tablet ea0cbd6bec674cf5831486f500cc7b36
1023 20:15:28.458117 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.458214 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2325123}
W20251023 20:15:28.458894 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
I20251023 20:15:28.459244 24349 tablet_replica.cc:333] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.459852 24349 raft_consensus.cc:2243] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.460078 24349 raft_consensus.cc:2272] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.461066 24349 ts_tablet_manager.cc:1916] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.463153 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8] alter complete (version 2)
I20251023 20:15:28.464533 24658 raft_consensus.cc:493] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.464679 24658 raft_consensus.cc:515] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.465240 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.465287 24658 leader_election.cc:290] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.468554 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.469558 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352812b56f8d4feb818a364d6adfd86d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:28.470309 23605 leader_election.cc:343] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.470916 24823 raft_consensus.cc:493] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.469765 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352812b56f8d4feb818a364d6adfd86d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.471866 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:28.472373 23604 leader_election.cc:343] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.472479 23604 leader_election.cc:304] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.472699 24655 raft_consensus.cc:2749] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.473603 24823 raft_consensus.cc:515] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.474228 24823 leader_election.cc:290] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:28.478122 23604 consensus_peers.cc:597] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.486505 24363 ts_tablet_manager.cc:1929] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.487219 24363 log.cc:1199] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/283166bbd8004ce3bca9f2ecbdbcbf48
I20251023 20:15:28.487493 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.487694 24363 ts_tablet_manager.cc:1950] T 283166bbd8004ce3bca9f2ecbdbcbf48 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.488449 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 94) took 2712 ms. Trace:
I20251023 20:15:28.488560 24363 rpcz_store.cc:276] 1023 20:15:25.775649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.775704 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:28.441670 (+2665966us) ts_tablet_manager.cc:1151] Deleting tablet 283166bbd8004ce3bca9f2ecbdbcbf48
1023 20:15:28.441680 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.465458 (+ 23778us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.486387 (+ 20929us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.488429 (+ 2042us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2665759}
I20251023 20:15:28.489058 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 283166bbd8004ce3bca9f2ecbdbcbf48 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:28.491539 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.493396 24363 tablet_replica.cc:333] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.493593 24363 raft_consensus.cc:2243] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.493934 24363 raft_consensus.cc:2272] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.494510 24363 ts_tablet_manager.cc:1916] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.495398 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.499667 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.501344 24349 ts_tablet_manager.cc:1929] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.501528 24349 log.cc:1199] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/2532c8567db14901b383f42e5f04b4c2
I20251023 20:15:28.501947 24349 ts_tablet_manager.cc:1950] T 2532c8567db14901b383f42e5f04b4c2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.503667 24659 raft_consensus.cc:493] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.503815 24659 raft_consensus.cc:515] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.504390 24659 leader_election.cc:290] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.505368 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.505553 23397 raft_consensus.cc:2468] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:28.505925 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.506065 23605 leader_election.cc:304] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:28.506328 24659 raft_consensus.cc:2804] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.506392 24659 raft_consensus.cc:493] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.506445 24659 raft_consensus.cc:3060] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.507717 23211 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 40) took 3345 ms. Trace:
I20251023 20:15:28.507813 23211 rpcz_store.cc:276] 1023 20:15:25.162485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.518604 (+356119us) service_pool.cc:225] Handling call
1023 20:15:25.518994 (+ 390us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:25.519306 (+ 312us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:25.519699 (+ 393us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:25.519901 (+ 202us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:25.940573 (+420672us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:28.507665 (+2567092us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:25.809211 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:25.809453 (+ 242us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:25.809458 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:25.809461 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:25.809651 (+ 190us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:25.809664 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:25.809669 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:25.809785 (+ 116us) tablet.cc:699] Row locks acquired
1023 20:15:25.809788 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:25.809844 (+ 56us) write_op.cc:270] Start()
1023 20:15:25.809874 (+ 30us) write_op.cc:276] Timestamp: P: 1761250525809777 usec, L: 0
1023 20:15:25.809933 (+ 59us) op_driver.cc:348] REPLICATION: starting
1023 20:15:25.810243 (+ 310us) log.cc:844] Serialized 405 byte log entry
1023 20:15:25.939886 (+129643us) op_driver.cc:464] REPLICATION: finished
1023 20:15:25.940040 (+ 154us) write_op.cc:301] APPLY: starting
1023 20:15:25.940057 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:25.940099 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:25.940130 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:25.940185 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:25.940217 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:25.940224 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:25.940299 (+ 75us) log.cc:844] Serialized 23 byte log entry
1023 20:15:25.940365 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.940383 (+ 18us) write_op.cc:454] Released schema lock
1023 20:15:25.940390 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:25.940462 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.940466 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":416,"apply.run_wall_time_us":416,"num_ops":1,"prepare.queue_time_us":288814,"prepare.run_cpu_time_us":945,"prepare.run_wall_time_us":124029,"raft.queue_time_us":122597,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":290,"replication_time_us":129922,"spinlock_wait_cycles":26880}]]}
I20251023 20:15:28.508939 24659 raft_consensus.cc:515] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.509063 23211 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_2e38cdb29ca54bdca4c030118988b786" } new_table_name: "default.Table_d22911da756748fb8939006df660e885" modify_external_catalogs: true
I20251023 20:15:28.510461 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:28.510637 23397 raft_consensus.cc:3060] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.513554 23397 raft_consensus.cc:2468] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:28.514037 23605 leader_election.cc:304] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:28.515174 24659 leader_election.cc:290] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.515372 24659 raft_consensus.cc:2804] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.515837 24659 raft_consensus.cc:697] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:28.516136 24659 consensus_queue.cc:237] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.505367 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.515029 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:28.518183 23604 leader_election.cc:343] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.518277 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 130) took 2385 ms. Trace:
I20251023 20:15:28.518347 24349 rpcz_store.cc:276] 1023 20:15:26.135208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.135440 (+ 232us) service_pool.cc:225] Handling call
1023 20:15:28.459193 (+2323753us) ts_tablet_manager.cc:1151] Deleting tablet 2532c8567db14901b383f42e5f04b4c2
1023 20:15:28.459203 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.482788 (+ 23585us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.501222 (+ 18434us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.518258 (+ 17036us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2323411}
W20251023 20:15:28.518484 23604 leader_election.cc:343] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.518770 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 2532c8567db14901b383f42e5f04b4c2 (table default.Table_bf99025eb142454fa0513fafcbe636aa [id=f52f1d83bd9345ffa08e6d3d282914e0]) successfully deleted
I20251023 20:15:28.520244 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 131) took 2384 ms. Trace:
I20251023 20:15:28.522866 24349 rpcz_store.cc:276] 1023 20:15:26.135916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.137298 (+ 1382us) service_pool.cc:225] Handling call
1023 20:15:28.520112 (+2382814us) ts_tablet_manager.cc:1151] Deleting tablet ac55e14745534a02a2f6a422cab6b149
1023 20:15:28.520125 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.520226 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2382540}
W20251023 20:15:28.525408 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ac55e14745534a02a2f6a422cab6b149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac55e14745534a02a2f6a422cab6b149 already in progress: creating tablet
I20251023 20:15:28.525900 24349 tablet_replica.cc:333] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.526333 24349 raft_consensus.cc:2243] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.526778 24349 raft_consensus.cc:2272] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.527756 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.528316 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.528782 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.529248 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.529682 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.530148 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.530586 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.531131 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.531559 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.532233 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.532424 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.532598 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.532651 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.532781 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.532848 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.532923 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.533437 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.533921 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.534355 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.534871 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.534905 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.535305 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.536311 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.536530 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.537163 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.537377 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.537654 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "252b5480222f49439052c86a4115cc9b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.537727 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.537781 23786 tablet_bootstrap.cc:492] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.537910 23786 ts_tablet_manager.cc:1403] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.222s user 0.010s sys 0.025s
I20251023 20:15:28.538782 24811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251023 20:15:28.538678 23786 raft_consensus.cc:359] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.539322 23786 raft_consensus.cc:385] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.539438 23786 raft_consensus.cc:740] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.539716 23786 consensus_queue.cc:260] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:28.542775 23216 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet af899c01b4af46d39d27f984cc3d5940 consensus state has no leader
I20251023 20:15:28.543279 24349 ts_tablet_manager.cc:1916] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.544618 24621 raft_consensus.cc:493] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:28.544770 24822 raft_consensus.cc:493] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.544751 24621 raft_consensus.cc:515] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.544888 24822 raft_consensus.cc:515] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.545359 24621 leader_election.cc:290] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.545536 24822 leader_election.cc:290] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.547017 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.547824 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.548631 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.548797 24780 tablet_replica.cc:333] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:28.548970 24780 raft_consensus.cc:2243] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.549171 24780 raft_consensus.cc:2272] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.550010 24780 ts_tablet_manager.cc:1916] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.550544 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.551116 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.551703 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.551761 23786 ts_tablet_manager.cc:1434] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251023 20:15:28.552212 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.552690 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.550014 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.553164 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.553717 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.554256 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.555461 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.556022 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.556110 23786 tablet_bootstrap.cc:492] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.556783 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.557402 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.557798 23786 tablet_bootstrap.cc:654] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.557976 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.558336 24780 ts_tablet_manager.cc:1929] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.558508 24780 log.cc:1199] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/7e08dad1d0f643efb3a821a1b9196290
I20251023 20:15:28.558565 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.559028 24780 ts_tablet_manager.cc:1950] T 7e08dad1d0f643efb3a821a1b9196290 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:28.559104 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.559633 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.560124 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.560616 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.561134 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.561638 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.561976 24659 raft_consensus.cc:493] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.562274 24259 raft_consensus.cc:493] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.562302 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.562268 24659 raft_consensus.cc:515] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.562415 24259 raft_consensus.cc:515] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.562889 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.562983 24259 leader_election.cc:290] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:28.563215 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca01bea2bb146ca8797a9592a840a81" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.563422 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.563722 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:28.563721 23605 leader_election.cc:343] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.564116 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2a27706ecdb4ab78bb96c10a9f6757f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.564558 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.564970 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca01bea2bb146ca8797a9592a840a81" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.565145 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:28.565498 23604 leader_election.cc:343] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.565599 23604 leader_election.cc:304] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.565707 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.565860 24658 raft_consensus.cc:2749] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.565891 24659 leader_election.cc:290] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.566504 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.567097 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.567658 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.568202 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.568552 24363 ts_tablet_manager.cc:1929] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.568660 24363 log.cc:1199] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:28.568930 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.568982 24363 ts_tablet_manager.cc:1950] T 66a2f3adb5d347a0adc198cd1ef93304 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.569614 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.569732 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 95) took 2793 ms. Trace:
I20251023 20:15:28.569816 24363 rpcz_store.cc:276] 1023 20:15:25.776048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.776090 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:28.493338 (+2717248us) ts_tablet_manager.cc:1151] Deleting tablet 66a2f3adb5d347a0adc198cd1ef93304
1023 20:15:28.493353 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.497670 (+ 4317us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.568430 (+ 70760us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.569713 (+ 1283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2717076}
I20251023 20:15:28.570317 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.570485 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 66a2f3adb5d347a0adc198cd1ef93304 (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]) successfully deleted
I20251023 20:15:28.570919 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.571466 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.571985 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.572475 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.572953 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.573501 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.573966 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.574038 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.574529 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.575114 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.575929 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.576478 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.577045 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.577493 24349 ts_tablet_manager.cc:1929] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.577687 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.577873 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 96) took 2800 ms. Trace:
I20251023 20:15:28.577960 24363 rpcz_store.cc:276] 1023 20:15:25.777468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.777514 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:28.577707 (+2800193us) ts_tablet_manager.cc:1151] Deleting tablet 419e41d7a37f4b888cb242b089c28790
1023 20:15:28.577721 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.577852 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2800023}
I20251023 20:15:28.578383 24363 tablet_replica.cc:333] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:28.578377 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 419e41d7a37f4b888cb242b089c28790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 419e41d7a37f4b888cb242b089c28790 already in progress: creating tablet
I20251023 20:15:28.578608 24363 raft_consensus.cc:2243] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.578639 24423 fs_manager.cc:730] Time spent opening block manager: real 0.265s user 0.018s sys 0.017s
I20251023 20:15:28.578990 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.579047 24363 raft_consensus.cc:2272] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.579592 24363 ts_tablet_manager.cc:1916] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.578895 24423 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data,/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
uuid: "ae5a310b4a514c0f812bda133c756062"
format_stamp: "Formatted at 2025-10-23 20:15:11 on dist-test-slave-g5bj"
I20251023 20:15:28.579942 24423 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251023 20:15:28.580142 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.580603 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.581043 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.581508 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.582044 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.582479 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:28.582739 23333 consensus_peers.cc:597] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe922c438a894a7aa04699b2d2897a28. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:28.583101 23333 leader_election.cc:343] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.578100 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.583432 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.583683 24349 log.cc:1199] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/2fa6a838b1bd409ba2a4e2074576ee23
I20251023 20:15:28.584030 24349 ts_tablet_manager.cc:1950] T 2fa6a838b1bd409ba2a4e2074576ee23 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.583995 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.584830 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.585304 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.586838 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 132) took 2449 ms. Trace:
I20251023 20:15:28.589785 24349 rpcz_store.cc:276] 1023 20:15:26.137870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.137949 (+ 79us) service_pool.cc:225] Handling call
1023 20:15:28.525842 (+2387893us) ts_tablet_manager.cc:1151] Deleting tablet 2fa6a838b1bd409ba2a4e2074576ee23
1023 20:15:28.525855 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.567392 (+ 41537us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.577374 (+ 9982us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.586817 (+ 9443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2387645}
I20251023 20:15:28.590708 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 2fa6a838b1bd409ba2a4e2074576ee23 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]) successfully deleted
I20251023 20:15:28.591904 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.592252 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.592228 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.592443 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.592469 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.592566 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.592715 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.592840 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.593295 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 133) took 2141 ms. Trace:
I20251023 20:15:28.593437 23471 leader_election.cc:304] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:28.593461 24349 rpcz_store.cc:276] 1023 20:15:26.452209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.452269 (+ 60us) service_pool.cc:225] Handling call
1023 20:15:28.593074 (+2140805us) ts_tablet_manager.cc:1151] Deleting tablet 1e19859a65a144598e1612be89daa3a1
1023 20:15:28.593134 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.593277 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2140579}
I20251023 20:15:28.593760 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.594045 24349 tablet_replica.cc:333] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:28.594125 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e19859a65a144598e1612be89daa3a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e19859a65a144598e1612be89daa3a1 already in progress: creating tablet
I20251023 20:15:28.594297 24349 raft_consensus.cc:2243] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.595031 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 6558265ff9e249c68123577958f1401a with delete_type TABLET_DATA_DELETED (replaced by tablet 2fea0afc8f63497d8b5cdf2352a20267) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.595945 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.596611 24349 raft_consensus.cc:2272] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.599623 24349 ts_tablet_manager.cc:1916] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.603163 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.606528 24363 ts_tablet_manager.cc:1929] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.607420 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97e2e0f839a4ef389fe100b6a81301e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.607717 24363 log.cc:1199] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:28.607756 23665 raft_consensus.cc:2468] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.608059 24363 ts_tablet_manager.cc:1950] T 951b2276d3324c2d999c8d7fb97767bc P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.608240 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.608810 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 97) took 2831 ms. Trace:
I20251023 20:15:28.609344 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.610841 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.611485 24349 ts_tablet_manager.cc:1929] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:28.611649 23790 tablet_bootstrap.cc:492] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.611815 23790 ts_tablet_manager.cc:1403] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.315s user 0.028s sys 0.033s
I20251023 20:15:28.611627 24363 rpcz_store.cc:276] 1023 20:15:25.777806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.777858 (+ 52us) service_pool.cc:225] Handling call
1023 20:15:28.578338 (+2800480us) ts_tablet_manager.cc:1151] Deleting tablet 951b2276d3324c2d999c8d7fb97767bc
1023 20:15:28.578348 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.582561 (+ 4213us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.606431 (+ 23870us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.608792 (+ 2361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2800347}
I20251023 20:15:28.612568 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 951b2276d3324c2d999c8d7fb97767bc (table default.Table_441bf8050be9453c85aef2cb7ac63ea4 [id=425e2178647a42aaa99cef01f06523f1]) successfully deleted
I20251023 20:15:28.612618 23790 raft_consensus.cc:359] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.612775 23790 raft_consensus.cc:385] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.612825 23790 raft_consensus.cc:740] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.614042 24363 tablet_replica.cc:333] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:28.614030 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
I20251023 20:15:28.614300 24363 raft_consensus.cc:2243] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:28.614372 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
I20251023 20:15:28.614537 24363 raft_consensus.cc:2272] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.615072 24363 ts_tablet_manager.cc:1916] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.615931 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2a27706ecdb4ab78bb96c10a9f6757f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.615979 23333 leader_election.cc:304] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.616761 24825 raft_consensus.cc:2749] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.612378 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.613319 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.617115 24349 log.cc:1199] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e5368813164949ff80e27bf59dcc931a
W20251023 20:15:28.617322 23333 consensus_peers.cc:597] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 283166bbd8004ce3bca9f2ecbdbcbf48. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.611042 23787 tablet_bootstrap.cc:492] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.617841 23787 ts_tablet_manager.cc:1403] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.323s user 0.026s sys 0.036s
I20251023 20:15:28.617826 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.618481 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.618659 24349 ts_tablet_manager.cc:1950] T e5368813164949ff80e27bf59dcc931a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:28.619840 23333 consensus_peers.cc:597] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a27f192792cf41288d69ca15ab538472. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.618492 23787 raft_consensus.cc:359] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.621217 23787 raft_consensus.cc:385] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.621268 23787 raft_consensus.cc:740] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.621526 23787 consensus_queue.cc:260] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.622141 23787 ts_tablet_manager.cc:1434] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:28.622833 24832 raft_consensus.cc:3060] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.624588 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.625032 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed46b7e67e4e4ae7b30ea69f1d3fdc2f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.625136 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97e2e0f839a4ef389fe100b6a81301e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.625260 23536 raft_consensus.cc:2468] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.625738 23790 consensus_queue.cc:260] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.627744 23790 ts_tablet_manager.cc:1434] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251023 20:15:28.628345 23787 tablet_bootstrap.cc:492] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.628804 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 136) took 2053 ms. Trace:
I20251023 20:15:28.629069 24349 rpcz_store.cc:276] 1023 20:15:26.575391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.575470 (+ 79us) service_pool.cc:225] Handling call
1023 20:15:28.593991 (+2018521us) ts_tablet_manager.cc:1151] Deleting tablet e5368813164949ff80e27bf59dcc931a
1023 20:15:28.594002 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.604260 (+ 10258us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.611387 (+ 7127us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.628784 (+ 17397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2016845}
W20251023 20:15:28.630342 23333 leader_election.cc:343] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:28.631589 24780 tablet_replica.cc:333] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:28.632972 24780 raft_consensus.cc:2243] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.633052 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.633368 24780 raft_consensus.cc:2272] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:28.632613 23470 consensus_peers.cc:597] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d82c92e863b2470699cc822b8cb8c14d. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.633952 24780 ts_tablet_manager.cc:1916] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.634024 24349 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 150) took 1267 ms. Trace:
I20251023 20:15:28.634085 24349 rpcz_store.cc:276] 1023 20:15:27.366989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:27.367957 (+ 968us) service_pool.cc:225] Handling call
1023 20:15:28.633907 (+1265950us) ts_tablet_manager.cc:1151] Deleting tablet af899c01b4af46d39d27f984cc3d5940
1023 20:15:28.633917 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.634007 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1262647}
I20251023 20:15:28.634181 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
W20251023 20:15:28.634936 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet af899c01b4af46d39d27f984cc3d5940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af899c01b4af46d39d27f984cc3d5940 already in progress: creating tablet
I20251023 20:15:28.635128 24349 tablet_replica.cc:333] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.638581 24825 raft_consensus.cc:493] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.638720 24825 raft_consensus.cc:515] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.639315 24825 leader_election.cc:290] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.640739 24349 raft_consensus.cc:2243] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.641039 24349 raft_consensus.cc:2272] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.641683 24349 ts_tablet_manager.cc:1916] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:28.641892 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a4f012469fa4e8e827e485287ad76dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a4f012469fa4e8e827e485287ad76dd already in progress: creating tablet
W20251023 20:15:28.642227 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d62dbf044acd4459b26d2e7e50f6206e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d62dbf044acd4459b26d2e7e50f6206e already in progress: creating tablet
W20251023 20:15:28.642491 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 377de4ea0bd6489284fac6b0fab52c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377de4ea0bd6489284fac6b0fab52c8c already in progress: creating tablet
W20251023 20:15:28.642781 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
I20251023 20:15:28.642876 24363 ts_tablet_manager.cc:1929] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.642995 24363 log.cc:1199] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/252b5480222f49439052c86a4115cc9b
W20251023 20:15:28.643121 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
I20251023 20:15:28.643236 24780 ts_tablet_manager.cc:1929] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.643272 24363 ts_tablet_manager.cc:1950] T 252b5480222f49439052c86a4115cc9b P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.643342 24780 log.cc:1199] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/f1a6fe726ef844829a7ffec9d4494637
W20251023 20:15:28.643424 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
I20251023 20:15:28.643679 24780 ts_tablet_manager.cc:1950] T f1a6fe726ef844829a7ffec9d4494637 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:28.631667 23333 leader_election.cc:304] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:28.643992 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 98) took 2865 ms. Trace:
I20251023 20:15:28.644057 24363 rpcz_store.cc:276] 1023 20:15:25.778104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.778141 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:28.613992 (+2835851us) ts_tablet_manager.cc:1151] Deleting tablet 252b5480222f49439052c86a4115cc9b
1023 20:15:28.614001 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.635433 (+ 21432us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.642762 (+ 7329us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.643971 (+ 1209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2835728}
I20251023 20:15:28.644409 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 99) took 2771 ms. Trace:
I20251023 20:15:28.644510 24363 rpcz_store.cc:276] 1023 20:15:25.872504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.872558 (+ 54us) service_pool.cc:225] Handling call
1023 20:15:28.644301 (+2771743us) ts_tablet_manager.cc:1151] Deleting tablet 5acdc66df9c748a1a6300962f62fcf36
1023 20:15:28.644310 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.644394 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2771559}
I20251023 20:15:28.644762 24363 tablet_replica.cc:333] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.644840 24822 raft_consensus.cc:2804] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.631073 23790 tablet_bootstrap.cc:492] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.644918 24822 raft_consensus.cc:493] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.644939 24363 raft_consensus.cc:2243] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.644985 24822 raft_consensus.cc:3060] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.644845 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.645552 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.645938 24830 raft_consensus.cc:493] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.646059 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.646601 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.647058 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.647500 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.647502 24822 raft_consensus.cc:515] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.647976 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.648110 24822 leader_election.cc:290] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.648506 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.648886 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 252b5480222f49439052c86a4115cc9b (table default.Table_bdb5b8891a814c998413f5b7a0a605cc [id=0674b989e3f74438b47e1fc653f5db49]) successfully deleted
I20251023 20:15:28.649586 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:28.653362 24363 raft_consensus.cc:2272] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.654726 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 7e08dad1d0f643efb3a821a1b9196290 (table default.Table_ee55921ad9e8422984b6afd71da0f4ee [id=c6c3d95bdfef4028a356f1365e9fa53d]) successfully deleted
I20251023 20:15:28.654817 24363 ts_tablet_manager.cc:1916] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.655050 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fea0afc8f63497d8b5cdf2352a20267" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.655222 23665 raft_consensus.cc:2468] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.656064 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fea0afc8f63497d8b5cdf2352a20267" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.656423 23471 leader_election.cc:304] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:28.656739 24836 raft_consensus.cc:2749] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.657992 24832 raft_consensus.cc:2749] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.659334 23790 tablet_bootstrap.cc:654] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.659812 23787 tablet_bootstrap.cc:654] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.661756 24349 ts_tablet_manager.cc:1929] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.663164 24349 log.cc:1199] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:28.663416 24363 ts_tablet_manager.cc:1929] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.663545 24349 ts_tablet_manager.cc:1950] T 95574277dde44983bb0a43d2b5ad01d5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.663553 24363 log.cc:1199] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:28.663918 24363 ts_tablet_manager.cc:1950] T 79e3d422a66f432789c0cf4a00e57af2 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.664311 24349 tablet_replica.cc:333] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.664484 24349 raft_consensus.cc:2243] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.664618 24349 pending_rounds.cc:70] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Trying to abort 1 pending ops.
I20251023 20:15:28.664661 24349 pending_rounds.cc:77] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082160193101824 op_type: NO_OP noop_request { }
I20251023 20:15:28.664749 24349 raft_consensus.cc:2889] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:28.664834 24349 raft_consensus.cc:2272] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.665038 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 100) took 2764 ms. Trace:
I20251023 20:15:28.665182 24363 rpcz_store.cc:276] 1023 20:15:25.900496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.900809 (+ 313us) service_pool.cc:225] Handling call
1023 20:15:28.644722 (+2743913us) ts_tablet_manager.cc:1151] Deleting tablet 79e3d422a66f432789c0cf4a00e57af2
1023 20:15:28.644731 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.660068 (+ 15337us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.663330 (+ 3262us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.665019 (+ 1689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2743753}
I20251023 20:15:28.665714 24363 tablet_replica.cc:333] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.665936 24363 raft_consensus.cc:2243] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:28.660465 23333 leader_election.cc:343] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.666147 24363 raft_consensus.cc:2272] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.666301 23333 leader_election.cc:304] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:28.666625 24822 raft_consensus.cc:2804] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:15:28.666739 24822 raft_consensus.cc:493] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:28.666863 24822 raft_consensus.cc:3060] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:28.666906 24363 ts_tablet_manager.cc:1916] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:28.667423 23333 consensus_peers.cc:597] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 283166bbd8004ce3bca9f2ecbdbcbf48. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:28.667641 23333 consensus_peers.cc:597] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fa6a838b1bd409ba2a4e2074576ee23. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:28.667840 23333 leader_election.cc:343] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:28.668066 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
I20251023 20:15:28.669292 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.669932 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.669983 24780 tablet_replica.cc:333] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:28.670270 24822 raft_consensus.cc:515] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.670785 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 79e3d422a66f432789c0cf4a00e57af2 (table default.Table_7b6d17b4f21740f0a5dadc35a23c919f [id=8b9eb030f1a24f68bb8e2e87b580727a]) successfully deleted
I20251023 20:15:28.670957 24780 raft_consensus.cc:2243] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.671118 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.671205 24780 raft_consensus.cc:2272] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.671342 24822 leader_election.cc:290] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.671512 24349 ts_tablet_manager.cc:1916] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.671775 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.672135 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.672472 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.672504 24780 ts_tablet_manager.cc:1916] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.672849 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.673226 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.673429 24363 ts_tablet_manager.cc:1929] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.673534 24363 log.cc:1199] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/f908f086946949829cac15418a584730
I20251023 20:15:28.673807 24363 ts_tablet_manager.cc:1950] T f908f086946949829cac15418a584730 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.673900 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.674749 23790 tablet_bootstrap.cc:492] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.674912 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 102) took 2773 ms. Trace:
I20251023 20:15:28.674945 23790 ts_tablet_manager.cc:1403] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20251023 20:15:28.675042 24363 rpcz_store.cc:276] 1023 20:15:25.901268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.901501 (+ 233us) service_pool.cc:225] Handling call
1023 20:15:28.665665 (+2764164us) ts_tablet_manager.cc:1151] Deleting tablet f908f086946949829cac15418a584730
1023 20:15:28.665676 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.670217 (+ 4541us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.673347 (+ 3130us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.674895 (+ 1548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2764023}
I20251023 20:15:28.675498 24822 raft_consensus.cc:493] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:28.675560 24363 tablet_replica.cc:333] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.675716 24363 raft_consensus.cc:2243] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.675734 24822 raft_consensus.cc:515] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.676043 24363 raft_consensus.cc:2272] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.676333 24822 leader_election.cc:290] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.677137 24363 ts_tablet_manager.cc:1916] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.677536 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.677175 23787 tablet_bootstrap.cc:492] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.677783 23787 ts_tablet_manager.cc:1403] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.002s
I20251023 20:15:28.678624 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.679242 23790 raft_consensus.cc:359] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.679529 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.679994 23536 raft_consensus.cc:3060] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.679559 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.679922 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.680653 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.680742 23790 raft_consensus.cc:385] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.680842 23790 raft_consensus.cc:740] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.679911 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.681942 23790 consensus_queue.cc:260] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.682195 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.682796 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97e2e0f839a4ef389fe100b6a81301e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:28.682943 23665 raft_consensus.cc:3060] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.685473 23665 raft_consensus.cc:2468] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:28.686295 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c97e2e0f839a4ef389fe100b6a81301e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.687980 23787 raft_consensus.cc:359] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.688369 23787 raft_consensus.cc:385] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.688566 23787 raft_consensus.cc:740] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.688997 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.689000 23787 consensus_queue.cc:260] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.689575 23790 ts_tablet_manager.cc:1434] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251023 20:15:28.690464 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet f1a6fe726ef844829a7ffec9d4494637 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]) successfully deleted
W20251023 20:15:28.690910 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:28.691278 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
I20251023 20:15:28.691936 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed46b7e67e4e4ae7b30ea69f1d3fdc2f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.692122 23532 raft_consensus.cc:3060] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 2
I20251023 20:15:28.692164 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.692929 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.694356 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.694526 23532 raft_consensus.cc:2468] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 2.
W20251023 20:15:28.695036 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
W20251023 20:15:28.695276 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
I20251023 20:15:28.695757 23790 tablet_bootstrap.cc:492] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.696611 23787 ts_tablet_manager.cc:1434] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251023 20:15:28.697328 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cec51c54ea534b1da4b1d170e79d9f1c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.697460 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.697497 23532 raft_consensus.cc:2468] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.697582 23787 tablet_bootstrap.cc:492] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.699302 23790 tablet_bootstrap.cc:654] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.700737 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:28.700356 23333 consensus_peers.cc:597] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 252b5480222f49439052c86a4115cc9b. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:28.701295 23333 leader_election.cc:304] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.701620 24822 raft_consensus.cc:2804] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.701705 24822 raft_consensus.cc:493] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.701808 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.701896 24822 raft_consensus.cc:3060] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.702675 23787 tablet_bootstrap.cc:654] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.704886 24822 raft_consensus.cc:515] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.704980 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.705981 24822 leader_election.cc:290] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:28.646080 24830 raft_consensus.cc:515] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:28.707751 23470 leader_election.cc:343] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a708cd011802458aa0e2991fd4b4a2d3
I20251023 20:15:28.706377 24349 ts_tablet_manager.cc:1929] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.706594 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.709192 24349 log.cc:1199] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/7e08dad1d0f643efb3a821a1b9196290
I20251023 20:15:28.709196 23333 leader_election.cc:304] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:28.709436 24822 raft_consensus.cc:2804] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.710131 24349 ts_tablet_manager.cc:1950] T 7e08dad1d0f643efb3a821a1b9196290 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.709764 24822 raft_consensus.cc:697] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.710373 24822 consensus_queue.cc:237] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.711229 24363 ts_tablet_manager.cc:1929] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.711400 24363 log.cc:1199] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/5e73ae95c73945f081c298f49d47c904
I20251023 20:15:28.711798 24363 ts_tablet_manager.cc:1950] T 5e73ae95c73945f081c298f49d47c904 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.711959 24349 tablet_replica.cc:333] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.712302 24349 raft_consensus.cc:2243] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:28.713769 23333 consensus_peers.cc:597] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a21ec49194e24723b31fd9c86f1f85e4. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:28.714003 23333 leader_election.cc:343] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ed46b7e67e4e4ae7b30ea69f1d3fdc2f
W20251023 20:15:28.714215 23333 leader_election.cc:343] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:28.714401 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 103) took 2813 ms. Trace:
I20251023 20:15:28.715104 24363 rpcz_store.cc:276] 1023 20:15:25.901363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.901665 (+ 302us) service_pool.cc:225] Handling call
1023 20:15:28.675514 (+2773849us) ts_tablet_manager.cc:1151] Deleting tablet 5e73ae95c73945f081c298f49d47c904
1023 20:15:28.675524 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.691739 (+ 16215us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.711122 (+ 19383us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.714379 (+ 3257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2773697}
I20251023 20:15:28.715909 24363 tablet_replica.cc:333] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.716470 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fea0afc8f63497d8b5cdf2352a20267" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:28.716634 23665 raft_consensus.cc:3060] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.716785 24363 raft_consensus.cc:2243] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.717064 24363 raft_consensus.cc:2272] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.717562 24363 ts_tablet_manager.cc:1916] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.719317 23665 raft_consensus.cc:2468] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:28.716282 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2fea0afc8f63497d8b5cdf2352a20267" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.721647 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.721642 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.721858 24349 raft_consensus.cc:2272] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:28.721993 23333 leader_election.cc:343] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.722417 24349 ts_tablet_manager.cc:1916] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.722469 24822 consensus_queue.cc:1048] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.722647 24821 raft_consensus.cc:493] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.722885 24821 raft_consensus.cc:515] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.723235 23333 leader_election.cc:304] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:28.723418 23333 raft_consensus.cc:2686] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251023 20:15:28.723471 24821 leader_election.cc:290] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.723677 23333 leader_election.cc:304] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:28.723891 24821 raft_consensus.cc:2804] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:15:28.723986 24821 raft_consensus.cc:493] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:28.724069 24821 raft_consensus.cc:3060] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:28.724881 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.726835 24363 ts_tablet_manager.cc:1929] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.726931 24363 log.cc:1199] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/4b6c3e7169154e3497dd3570dca0d745
I20251023 20:15:28.727421 24821 raft_consensus.cc:515] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.728051 24821 leader_election.cc:290] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:28.728444 23333 leader_election.cc:343] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.729522 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.732543 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.734207 24349 ts_tablet_manager.cc:1929] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.734490 23790 tablet_bootstrap.cc:492] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.734723 23790 ts_tablet_manager.cc:1403] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20251023 20:15:28.735638 23790 raft_consensus.cc:359] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.735924 23790 raft_consensus.cc:385] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.736033 23790 raft_consensus.cc:740] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.736348 23790 consensus_queue.cc:260] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.737779 23790 ts_tablet_manager.cc:1434] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:28.739861 23790 tablet_bootstrap.cc:492] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.743243 23333 leader_election.cc:304] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.744192 23790 tablet_bootstrap.cc:654] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.745484 24821 raft_consensus.cc:2804] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.745620 24821 raft_consensus.cc:697] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.745883 24821 consensus_queue.cc:237] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.746640 24363 ts_tablet_manager.cc:1950] T 4b6c3e7169154e3497dd3570dca0d745 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.732693 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.738543 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.741348 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.747018 23665 raft_consensus.cc:2468] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.741724 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.743489 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.747366 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 104) took 2845 ms. Trace:
I20251023 20:15:28.747479 24363 rpcz_store.cc:276] 1023 20:15:25.901461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.901845 (+ 384us) service_pool.cc:225] Handling call
1023 20:15:28.715860 (+2814015us) ts_tablet_manager.cc:1151] Deleting tablet 4b6c3e7169154e3497dd3570dca0d745
1023 20:15:28.715871 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.724056 (+ 8185us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.726739 (+ 2683us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.747351 (+ 20612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2813873}
I20251023 20:15:28.749814 24363 tablet_replica.cc:333] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.750113 24363 raft_consensus.cc:2243] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.750537 24363 raft_consensus.cc:2272] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.751315 24363 ts_tablet_manager.cc:1916] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:28.751926 23217 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 163) took 3320 ms (client timeout 1000 ms). Trace:
W20251023 20:15:28.753355 23333 leader_election.cc:343] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:28.753595 23333 leader_election.cc:304] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:28.754714 24822 raft_consensus.cc:2804] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.754761 23787 tablet_bootstrap.cc:492] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.754838 24822 raft_consensus.cc:493] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.754874 23787 ts_tablet_manager.cc:1403] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.001s
I20251023 20:15:28.754921 24822 raft_consensus.cc:3060] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.755597 24659 raft_consensus.cc:493] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.755620 23787 raft_consensus.cc:359] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.755770 23787 raft_consensus.cc:385] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.755815 23787 raft_consensus.cc:740] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.755723 24659 raft_consensus.cc:515] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.756017 23787 consensus_queue.cc:260] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.756304 24659 leader_election.cc:290] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.757661 24822 raft_consensus.cc:515] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.757838 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0150ab3778c34c77a7e772dd6e82b32f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.758311 23787 ts_tablet_manager.cc:1434] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251023 20:15:28.758458 23605 leader_election.cc:343] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.758590 23787 tablet_bootstrap.cc:492] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.758513 24822 leader_election.cc:290] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.759341 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.760066 23787 tablet_bootstrap.cc:654] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.760331 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:28.760574 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.761174 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.761652 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.762148 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.763056 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.763317 24780 ts_tablet_manager.cc:1929] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.763523 24780 log.cc:1199] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:28.763631 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.763947 24780 ts_tablet_manager.cc:1950] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:28.764732 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.765264 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.765787 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.766243 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.766687 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.767192 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.767627 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.768623 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.769515 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.770042 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.770552 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.771375 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.771819 23787 tablet_bootstrap.cc:492] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.771953 23787 ts_tablet_manager.cc:1403] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20251023 20:15:28.771937 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.772462 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.772774 23787 raft_consensus.cc:359] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.772940 23787 raft_consensus.cc:385] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.772953 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.772990 23787 raft_consensus.cc:740] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.773514 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.774103 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.774569 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.736955 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.737836 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.774986 24349 log.cc:1199] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:28.774814 24822 raft_consensus.cc:493] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.775205 24822 raft_consensus.cc:515] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.775324 24349 ts_tablet_manager.cc:1950] T ed46b7e67e4e4ae7b30ea69f1d3fdc2f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.775974 24822 leader_election.cc:290] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.738114 24830 leader_election.cc:290] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:28.742415 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cec51c54ea534b1da4b1d170e79d9f1c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.776320 23529 raft_consensus.cc:3060] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 2
I20251023 20:15:28.776577 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.777107 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.777591 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.777942 23787 consensus_queue.cc:260] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.778143 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.778497 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.778800 23529 raft_consensus.cc:2468] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 2.
I20251023 20:15:28.746300 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.746599 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.752554 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.753957 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.757493 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0150ab3778c34c77a7e772dd6e82b32f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.779193 24659 raft_consensus.cc:493] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.779304 24659 raft_consensus.cc:515] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.779433 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:28.779529 23604 leader_election.cc:343] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.779631 23604 leader_election.cc:304] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.779719 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.779850 24659 leader_election.cc:290] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.779973 24659 raft_consensus.cc:2749] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.780054 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet e5368813164949ff80e27bf59dcc931a with delete_type TABLET_DATA_DELETED (replaced by tablet abdad7d76eb8441496d38642382f74bb) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.780339 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a04c65155ce048cb82b150cbe20be849" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.780449 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:28.780738 23604 leader_election.cc:343] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.781131 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.774590 23786 tablet_bootstrap.cc:492] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.781314 23786 ts_tablet_manager.cc:1403] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.225s user 0.017s sys 0.033s
I20251023 20:15:28.781734 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:28.781898 23665 raft_consensus.cc:3060] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.782095 23786 raft_consensus.cc:359] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.782241 23786 raft_consensus.cc:385] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.782284 23786 raft_consensus.cc:740] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.782476 23786 consensus_queue.cc:260] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.782912 23786 ts_tablet_manager.cc:1434] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:28.783346 23786 tablet_bootstrap.cc:492] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.784386 23665 raft_consensus.cc:2468] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:28.785529 23786 tablet_bootstrap.cc:654] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.786320 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a04c65155ce048cb82b150cbe20be849" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.786573 23787 ts_tablet_manager.cc:1434] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251023 20:15:28.786830 24349 tablet_replica.cc:333] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:28.786824 23605 leader_election.cc:343] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.786921 23605 leader_election.cc:304] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.786984 24349 raft_consensus.cc:2243] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.787118 24349 raft_consensus.cc:2272] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.787248 24659 raft_consensus.cc:2749] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.787757 24349 ts_tablet_manager.cc:1916] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.790655 23790 tablet_bootstrap.cc:492] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.790869 23790 ts_tablet_manager.cc:1403] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.000s
I20251023 20:15:28.791924 24659 raft_consensus.cc:493] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.792053 24659 raft_consensus.cc:515] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.792604 24659 leader_election.cc:290] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:28.793529 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.793534 23790 raft_consensus.cc:359] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.793717 23790 raft_consensus.cc:385] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.793820 23790 raft_consensus.cc:740] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.793974 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.794137 23790 consensus_queue.cc:260] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:28.794472 23605 leader_election.cc:343] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:28.794631 23604 leader_election.cc:343] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.794737 23604 leader_election.cc:304] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.795169 24658 raft_consensus.cc:2749] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.795511 23790 ts_tablet_manager.cc:1434] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:28.795998 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.796420 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "600f0110f8bd4b05983d5922d871c266" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.796607 23665 raft_consensus.cc:2468] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.797181 23787 tablet_bootstrap.cc:492] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.797427 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "600f0110f8bd4b05983d5922d871c266" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.799263 23787 tablet_bootstrap.cc:654] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.800984 24363 ts_tablet_manager.cc:1929] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.801179 24363 log.cc:1199] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/2fa6a838b1bd409ba2a4e2074576ee23
I20251023 20:15:28.801630 24363 ts_tablet_manager.cc:1950] T 2fa6a838b1bd409ba2a4e2074576ee23 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.802776 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 101) took 2901 ms. Trace:
I20251023 20:15:28.802929 24363 rpcz_store.cc:276] 1023 20:15:25.901188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.902109 (+ 921us) service_pool.cc:225] Handling call
1023 20:15:28.749761 (+2847652us) ts_tablet_manager.cc:1151] Deleting tablet 2fa6a838b1bd409ba2a4e2074576ee23
1023 20:15:28.749772 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.770047 (+ 20275us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.800859 (+ 30812us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.802755 (+ 1896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2847534}
I20251023 20:15:28.803570 24363 tablet_replica.cc:333] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.803822 24363 raft_consensus.cc:2243] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.804126 24363 raft_consensus.cc:2272] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:28.804207 23333 leader_election.cc:343] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.804593 23333 leader_election.cc:304] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:28.804986 24363 ts_tablet_manager.cc:1916] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.805411 24821 raft_consensus.cc:2804] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:15:28.806303 24821 raft_consensus.cc:697] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 2 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.806841 24821 consensus_queue.cc:237] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.808975 24349 ts_tablet_manager.cc:1929] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.809156 24349 log.cc:1199] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:28.809561 24349 ts_tablet_manager.cc:1950] T cb915d114fc0421999d079c23cad8eb5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:28.810809 23333 leader_election.cc:343] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.754418 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 95574277dde44983bb0a43d2b5ad01d5 (table default.Table_2d1938985e9345eba1d4fb750b552e78 [id=8b537204d252421c8ef4084d1dd9226f]) successfully deleted
W20251023 20:15:28.788349 23214 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 189) took 3322 ms (client timeout 1000 ms). Trace:
W20251023 20:15:28.811367 23204 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 57097us
I20251023 20:15:28.811647 23787 tablet_bootstrap.cc:492] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.811796 23787 ts_tablet_manager.cc:1403] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
W20251023 20:15:28.811797 23333 leader_election.cc:343] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:28.811784 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
W20251023 20:15:28.812134 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
I20251023 20:15:28.812251 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a708cd011802458aa0e2991fd4b4a2d3" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.812383 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 7e08dad1d0f643efb3a821a1b9196290 (table default.Table_ee55921ad9e8422984b6afd71da0f4ee [id=c6c3d95bdfef4028a356f1365e9fa53d]) successfully deleted
I20251023 20:15:28.812631 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f (table default.Table_7b49c2bed47942a1b7cb25f8675a69ea [id=8cf42619d4b44a5990f7f63f1e1315ed]) successfully deleted
I20251023 20:15:28.812848 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:28.812929 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
I20251023 20:15:28.813241 24349 tablet_replica.cc:333] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:28.813268 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f1a6fe726ef844829a7ffec9d4494637 because the tablet was not found. No further retry: Not found: Tablet not found: f1a6fe726ef844829a7ffec9d4494637
I20251023 20:15:28.813436 24349 raft_consensus.cc:2243] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.813561 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.813532 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet cb915d114fc0421999d079c23cad8eb5 (table default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]) successfully deleted
I20251023 20:15:28.813650 24349 raft_consensus.cc:2272] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:28.813896 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 48436f3d8f1443db990a2d10b668b70a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48436f3d8f1443db990a2d10b668b70a already in progress: creating tablet
I20251023 20:15:28.813993 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.814148 24349 ts_tablet_manager.cc:1916] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.814165 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet f908f086946949829cac15418a584730 (table default.Table_a1c9b5357e504ba2a116c58ff86ad13c [id=25a11be83f484fa6abecb21ce1145e1e]) successfully deleted
I20251023 20:15:28.814425 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.814553 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 5e73ae95c73945f081c298f49d47c904 (table default.Table_8d8ada20a7bb43eb9a534aedca23cfd2 [id=6fb3e731496a466a918ab84247724a22]) successfully deleted
I20251023 20:15:28.814828 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 4b6c3e7169154e3497dd3570dca0d745 (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]) successfully deleted
I20251023 20:15:28.814826 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.815237 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:28.815290 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 2fa6a838b1bd409ba2a4e2074576ee23 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]) successfully deleted
I20251023 20:15:28.815922 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.816448 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.816932 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.817430 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:28.817742 23205 catalog_manager.cc:5058] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): alter failed for tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]),no further retry: Not found: Tablet not found: e5368813164949ff80e27bf59dcc931a
I20251023 20:15:28.817909 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.818400 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.818843 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.820626 23536 raft_consensus.cc:3060] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.821391 23787 raft_consensus.cc:359] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.821566 23787 raft_consensus.cc:385] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.821622 23787 raft_consensus.cc:740] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:28.811367 23214 rpcz_store.cc:269] 1023 20:15:25.465704 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.465886 (+ 182us) service_pool.cc:225] Handling call
1023 20:15:26.486464 (+1020578us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.486473 (+ 9us) write_op.cc:454] Released schema lock
1023 20:15:28.788284 (+2301811us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.452904 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.453026 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.453033 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:26.453035 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:26.453169 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.453182 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.453187 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.453293 (+ 106us) tablet.cc:699] Row locks acquired
1023 20:15:26.453295 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:26.453336 (+ 41us) write_op.cc:270] Start()
1023 20:15:26.453372 (+ 36us) write_op.cc:276] Timestamp: P: 1761250526453287 usec, L: 0
1023 20:15:26.453376 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.453589 (+ 213us) log.cc:844] Serialized 919 byte log entry
1023 20:15:26.457228 (+ 3639us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.457379 (+ 151us) write_op.cc:301] APPLY: starting
1023 20:15:26.457396 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.457447 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.457477 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.457554 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.457586 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:26.457593 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:26.457686 (+ 93us) log.cc:844] Serialized 29 byte log entry
1023 20:15:26.457747 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.457771 (+ 24us) write_op.cc:454] Released schema lock
1023 20:15:26.457778 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":120524,"reported_tablets":239,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":451,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":804,"prepare.run_wall_time_us":805,"raft.queue_time_us":1112,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":233,"replication_time_us":3801}]]}
I20251023 20:15:28.821849 23787 consensus_queue.cc:260] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:28.811403 23217 rpcz_store.cc:269] 1023 20:15:25.431285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.463400 (+ 32115us) service_pool.cc:225] Handling call
1023 20:15:28.751865 (+3288465us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.324721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.324825 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.324830 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:26.324832 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:26.324912 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.324923 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.324925 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:26.324979 (+ 54us) tablet.cc:699] Row locks acquired
1023 20:15:26.324980 (+ 1us) write_op.cc:260] PREPARE: finished
1023 20:15:26.325009 (+ 29us) write_op.cc:270] Start()
1023 20:15:26.325040 (+ 31us) write_op.cc:276] Timestamp: P: 1761250526324972 usec, L: 0
1023 20:15:26.325043 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.325238 (+ 195us) log.cc:844] Serialized 536 byte log entry
1023 20:15:26.344983 (+ 19745us) op_driver.cc:464] REPLICATION: finished
1023 20:15:26.345155 (+ 172us) write_op.cc:301] APPLY: starting
1023 20:15:26.345175 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:26.345223 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:26.345248 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:26.345314 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:26.345349 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:26.345357 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:26.345450 (+ 93us) log.cc:844] Serialized 23 byte log entry
1023 20:15:26.345515 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.345540 (+ 25us) write_op.cc:454] Released schema lock
1023 20:15:26.345547 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:26.345650 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:26.345654 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":122188,"reported_tablets":258,"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":460,"num_ops":1,"prepare.queue_time_us":209,"prepare.run_cpu_time_us":639,"prepare.run_wall_time_us":640,"raft.queue_time_us":811,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":244,"replication_time_us":19903}]]}
I20251023 20:15:28.827209 23787 ts_tablet_manager.cc:1434] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251023 20:15:28.829509 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.830083 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.830518 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.830953 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.831460 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.831961 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.832221 23787 tablet_bootstrap.cc:492] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.832907 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.833559 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.833781 23787 tablet_bootstrap.cc:654] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.834179 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.834632 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.835614 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.836071 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.836804 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.837297 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.837776 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.837805 24363 ts_tablet_manager.cc:1929] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.837903 24363 log.cc:1199] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/1a64a7c2e1f242e1b54243489622bae5
I20251023 20:15:28.838269 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.838798 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.839236 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.839653 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.840102 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.840514 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.840970 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.841554 23471 leader_election.cc:304] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:28.841958 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.842039 24830 raft_consensus.cc:2749] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.842334 24363 ts_tablet_manager.cc:1950] T 1a64a7c2e1f242e1b54243489622bae5 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.843101 23790 tablet_bootstrap.cc:492] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.843178 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 109) took 2940 ms. Trace:
I20251023 20:15:28.842794 23333 leader_election.cc:304] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.843252 24363 rpcz_store.cc:276] 1023 20:15:25.902666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.902709 (+ 43us) service_pool.cc:225] Handling call
1023 20:15:28.803500 (+2900791us) ts_tablet_manager.cc:1151] Deleting tablet 1a64a7c2e1f242e1b54243489622bae5
1023 20:15:28.803512 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.834667 (+ 31155us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.837698 (+ 3031us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.843154 (+ 5456us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2900626}
I20251023 20:15:28.843842 24822 raft_consensus.cc:2804] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.844036 24822 raft_consensus.cc:697] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.844092 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 108) took 2941 ms. Trace:
I20251023 20:15:28.844156 24363 rpcz_store.cc:276] 1023 20:15:25.902576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.902903 (+ 327us) service_pool.cc:225] Handling call
1023 20:15:28.843953 (+2941050us) ts_tablet_manager.cc:1151] Deleting tablet bb475e303d07491bbc19dc3f628bceb7
1023 20:15:28.843964 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.844074 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2940925}
I20251023 20:15:28.844328 24822 consensus_queue.cc:237] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.844985 23790 tablet_bootstrap.cc:654] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.847352 23787 tablet_bootstrap.cc:492] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.847502 23787 ts_tablet_manager.cc:1403] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251023 20:15:28.849009 24780 tablet_replica.cc:333] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:28.849327 24780 raft_consensus.cc:2243] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.849561 24780 raft_consensus.cc:2272] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.850095 23787 raft_consensus.cc:359] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.850200 24780 ts_tablet_manager.cc:1916] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.850281 23787 raft_consensus.cc:385] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.850334 23787 raft_consensus.cc:740] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.850576 23787 consensus_queue.cc:260] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:28.790796 23215 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 164) took 2228 ms (client timeout 1000 ms). Trace:
W20251023 20:15:28.852962 23215 rpcz_store.cc:269] 1023 20:15:26.562308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.566716 (+ 4408us) service_pool.cc:225] Handling call
1023 20:15:28.790735 (+2224019us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:28.327353 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.327464 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.327470 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:28.327472 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:28.327581 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.327594 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.327598 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.327660 (+ 62us) tablet.cc:699] Row locks acquired
1023 20:15:28.327662 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.327696 (+ 34us) write_op.cc:270] Start()
1023 20:15:28.327726 (+ 30us) write_op.cc:276] Timestamp: P: 1761250528327659 usec, L: 0
1023 20:15:28.327730 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.327894 (+ 164us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.348032 (+ 20138us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.348181 (+ 149us) write_op.cc:301] APPLY: starting
1023 20:15:28.348199 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.348243 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.348278 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.348336 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.348370 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.348376 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:28.348466 (+ 90us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.348550 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.348572 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:28.348580 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.348659 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.348663 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:28.351041 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.351164 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.351170 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:28.351173 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:28.351276 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.351288 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.351293 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.351359 (+ 66us) tablet.cc:699] Row locks acquired
1023 20:15:28.351361 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.351392 (+ 31us) write_op.cc:270] Start()
1023 20:15:28.351422 (+ 30us) write_op.cc:276] Timestamp: P: 1761250528351355 usec, L: 0
1023 20:15:28.351425 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.351591 (+ 166us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.358213 (+ 6622us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.358360 (+ 147us) write_op.cc:301] APPLY: starting
1023 20:15:28.358389 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.358428 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.358452 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.358506 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.358534 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.358541 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:28.358611 (+ 70us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.358662 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.358679 (+ 17us) write_op.cc:454] Released schema lock
1023 20:15:28.358684 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.358741 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.358745 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:28.395497 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.395627 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.395634 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.395637 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:28.395766 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.395791 (+ 25us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.395796 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.395864 (+ 68us) tablet.cc:699] Row locks acquired
1023 20:15:28.395866 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.395909 (+ 43us) write_op.cc:270] Start()
1023 20:15:28.395948 (+ 39us) write_op.cc:276] Timestamp: P: 1761250528395861 usec, L: 0
1023 20:15:28.395951 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.396135 (+ 184us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.401530 (+ 5395us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.401648 (+ 118us) write_op.cc:301] APPLY: starting
1023 20:15:28.401665 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.401708 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.402624 (+ 916us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.402698 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.402735 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.402743 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:28.402842 (+ 99us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.402913 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.402936 (+ 23us) write_op.cc:454] Released schema lock
1023 20:15:28.402944 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.403018 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.403023 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":130095,"reported_tablets":256,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":469,"apply.run_wall_time_us":468,"num_ops":1,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":646,"raft.queue_time_us":192,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":204,"replication_time_us":20268}],["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":375,"apply.run_wall_time_us":374,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":652,"prepare.run_wall_time_us":652,"raft.queue_time_us":63,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":6755}],["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":497,"apply.run_wall_time_us":1361,"num_ops":1,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":737,"raft.queue_time_us":652,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":378,"replication_time_us":5545}]]}
I20251023 20:15:28.854022 23333 leader_election.cc:304] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.854959 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_dce798c03c814b1899812dc8c06f4452"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:28.855455 24670 raft_consensus.cc:2804] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.855600 24670 raft_consensus.cc:493] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.855695 24670 raft_consensus.cc:3060] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.855819 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:28.856875 23787 ts_tablet_manager.cc:1434] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
W20251023 20:15:28.857170 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dce798c03c814b1899812dc8c06f4452 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:28.802939 23216 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 190) took 4353 ms (client timeout 1000 ms). Trace:
W20251023 20:15:28.801561 23212 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 191) took 3252 ms (client timeout 1000 ms). Trace:
I20251023 20:15:28.858317 24670 raft_consensus.cc:515] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:28.858335 23212 rpcz_store.cc:269] 1023 20:15:25.548901 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.822616 (+273715us) service_pool.cc:225] Handling call
1023 20:15:28.801507 (+2978891us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:26.995600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:26.995743 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:26.995751 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:26.995754 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:26.995904 (+ 150us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:26.995919 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:26.995924 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:26.996037 (+ 113us) tablet.cc:699] Row locks acquired
1023 20:15:26.996040 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:26.996082 (+ 42us) write_op.cc:270] Start()
1023 20:15:26.996120 (+ 38us) write_op.cc:276] Timestamp: P: 1761250526996031 usec, L: 0
1023 20:15:26.996124 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:26.996341 (+ 217us) log.cc:844] Serialized 919 byte log entry
1023 20:15:27.004120 (+ 7779us) op_driver.cc:464] REPLICATION: finished
1023 20:15:27.004239 (+ 119us) write_op.cc:301] APPLY: starting
1023 20:15:27.004255 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:27.004303 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:27.004358 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:27.004445 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:27.004480 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:27.004488 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:27.004591 (+ 103us) log.cc:844] Serialized 29 byte log entry
1023 20:15:27.004654 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:27.004678 (+ 24us) write_op.cc:454] Released schema lock
1023 20:15:27.004684 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:27.004761 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:27.004765 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":140486,"reported_tablets":245,"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":512,"apply.run_wall_time_us":512,"num_ops":2,"prepare.queue_time_us":23031,"prepare.run_cpu_time_us":861,"prepare.run_wall_time_us":861,"raft.queue_time_us":3017,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":257,"replication_time_us":7964}]]}
I20251023 20:15:28.858969 24670 leader_election.cc:290] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.859025 23212 master_service.cc:954] ReplaceTablet: received request to replace tablet d2ae19cc80194363b8a9c96fbafb4ebe from {username='slave'} at 127.0.0.1:55414
W20251023 20:15:28.859799 23333 consensus_peers.cc:597] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a708cd011802458aa0e2991fd4b4a2d3. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:28.860014 23216 rpcz_store.cc:269] 1023 20:15:24.449380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:24.454376 (+ 4996us) service_pool.cc:225] Handling call
1023 20:15:25.025836 (+571460us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.025843 (+ 7us) write_op.cc:454] Released schema lock
1023 20:15:28.802888 (+3777045us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:25.017459 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:25.017620 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:25.017628 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:25.017630 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:25.017834 (+ 204us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:25.017851 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:25.017857 (+ 6us) tablet.cc:683] Acquiring locks for 5 operations
1023 20:15:25.018015 (+ 158us) tablet.cc:699] Row locks acquired
1023 20:15:25.018018 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:25.018058 (+ 40us) write_op.cc:270] Start()
1023 20:15:25.018098 (+ 40us) write_op.cc:276] Timestamp: P: 1761250525018036 usec, L: 0
1023 20:15:25.018102 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:25.018313 (+ 211us) log.cc:844] Serialized 2069 byte log entry
1023 20:15:25.022502 (+ 4189us) op_driver.cc:464] REPLICATION: finished
1023 20:15:25.023034 (+ 532us) write_op.cc:301] APPLY: starting
1023 20:15:25.023055 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:25.023120 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:25.023148 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:25.023266 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:25.023306 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
1023 20:15:25.023315 (+ 9us) write_op.cc:312] APPLY: finished
1023 20:15:25.023408 (+ 93us) log.cc:844] Serialized 47 byte log entry
1023 20:15:25.023471 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:25.023500 (+ 29us) write_op.cc:454] Released schema lock
1023 20:15:25.023507 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":21045,"reported_tablets":241,"child_traces":[["op",{"apply.queue_time_us":466,"apply.run_cpu_time_us":535,"apply.run_wall_time_us":536,"num_ops":5,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":972,"prepare.run_wall_time_us":1289,"raft.queue_time_us":39,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":253,"replication_time_us":4365}]]}
I20251023 20:15:28.861064 23216 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_bdcbe2f8128a4a89b305439e5123c2fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:28.861773 23216 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdcbe2f8128a4a89b305439e5123c2fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:28.862154 23787 tablet_bootstrap.cc:492] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.862403 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f (table default.Table_7b49c2bed47942a1b7cb25f8675a69ea [id=8cf42619d4b44a5990f7f63f1e1315ed]) successfully deleted
I20251023 20:15:28.862900 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:28.863013 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
I20251023 20:15:28.863631 23787 tablet_bootstrap.cc:654] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.863782 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "600f0110f8bd4b05983d5922d871c266" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:28.863945 23665 raft_consensus.cc:3060] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.864408 24363 tablet_replica.cc:333] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.864571 24363 raft_consensus.cc:2243] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.864759 24363 pending_rounds.cc:70] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Trying to abort 1 pending ops.
I20251023 20:15:28.864802 24363 pending_rounds.cc:77] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082162852708352 op_type: NO_OP noop_request { }
I20251023 20:15:28.864872 24363 raft_consensus.cc:2889] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:28.864933 24363 raft_consensus.cc:2272] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.865546 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.866191 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.866422 23665 raft_consensus.cc:2468] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:28.867475 24363 ts_tablet_manager.cc:1916] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.868012 24349 ts_tablet_manager.cc:1929] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.868139 24349 log.cc:1199] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/aa794b3bec4b4b3993c814ea7cd02ee2
I20251023 20:15:28.868404 24349 ts_tablet_manager.cc:1950] T aa794b3bec4b4b3993c814ea7cd02ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.871068 24349 tablet_replica.cc:333] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.871423 24349 raft_consensus.cc:2243] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.871963 24349 raft_consensus.cc:2272] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.873178 24349 ts_tablet_manager.cc:1916] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.873915 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.874426 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.874873 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.875294 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.875713 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.876142 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.876577 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.877072 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet c95db7b5b9fd46acae8ffa7119915907 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:15 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.877557 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.878021 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.878435 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.879545 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.879985 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.880403 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.880885 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.881332 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 33b5efb30b5c491c8eafe3ac25688843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.881570 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "600f0110f8bd4b05983d5922d871c266" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.881788 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.882172 23787 tablet_bootstrap.cc:492] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.882313 23787 ts_tablet_manager.cc:1403] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20251023 20:15:28.883109 23787 raft_consensus.cc:359] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.883270 23787 raft_consensus.cc:385] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.883318 23787 raft_consensus.cc:740] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:28.884343 23333 leader_election.cc:343] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.883538 23787 consensus_queue.cc:260] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.885859 23790 tablet_bootstrap.cc:492] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.886061 23790 ts_tablet_manager.cc:1403] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20251023 20:15:28.885936 24363 ts_tablet_manager.cc:1929] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.886910 23790 raft_consensus.cc:359] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.887015 24363 log.cc:1199] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:28.887163 23790 raft_consensus.cc:385] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.887354 23790 raft_consensus.cc:740] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.887607 23790 consensus_queue.cc:260] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.888397 24363 ts_tablet_manager.cc:1950] T 1ae910ac41fd4e349f159ab27e44375a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.888767 23790 ts_tablet_manager.cc:1434] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:28.889103 23790 tablet_bootstrap.cc:492] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.889313 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 106) took 2986 ms. Trace:
I20251023 20:15:28.889932 24659 raft_consensus.cc:493] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.890389 24363 rpcz_store.cc:276] 1023 20:15:25.902441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.903049 (+ 608us) service_pool.cc:225] Handling call
1023 20:15:28.864364 (+2961315us) ts_tablet_manager.cc:1151] Deleting tablet 1ae910ac41fd4e349f159ab27e44375a
1023 20:15:28.864372 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.878141 (+ 13769us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.885825 (+ 7684us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.889293 (+ 3468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2961204}
I20251023 20:15:28.891237 23790 tablet_bootstrap.cc:654] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.890394 24659 raft_consensus.cc:515] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.893699 24659 leader_election.cc:290] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.894430 23333 leader_election.cc:304] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.894552 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "419e41d7a37f4b888cb242b089c28790" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:28.894778 24670 raft_consensus.cc:2804] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.894878 24670 raft_consensus.cc:697] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.895030 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "419e41d7a37f4b888cb242b089c28790" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.895102 24670 consensus_queue.cc:237] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:28.895501 23604 leader_election.cc:343] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:28.895713 23605 leader_election.cc:343] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.895804 23605 leader_election.cc:304] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.896245 24659 raft_consensus.cc:2749] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:28.898324 24363 tablet_replica.cc:333] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.898586 24363 raft_consensus.cc:2243] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.898811 24363 raft_consensus.cc:2272] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.900660 23786 tablet_bootstrap.cc:492] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:28.900892 23786 ts_tablet_manager.cc:1403] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.012s
I20251023 20:15:28.901845 23786 raft_consensus.cc:359] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.902079 23786 raft_consensus.cc:385] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.902166 23786 raft_consensus.cc:740] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.902408 23786 consensus_queue.cc:260] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:28.904588 23787 ts_tablet_manager.cc:1434] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251023 20:15:28.905102 24363 ts_tablet_manager.cc:1916] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.907375 23786 ts_tablet_manager.cc:1434] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:28.907704 23786 tablet_bootstrap.cc:492] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:28.907889 24822 raft_consensus.cc:493] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.908073 24822 raft_consensus.cc:515] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.908794 24822 leader_election.cc:290] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.914047 23787 tablet_bootstrap.cc:492] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.915213 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48436f3d8f1443db990a2d10b668b70a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.915380 23665 raft_consensus.cc:2468] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.917114 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48436f3d8f1443db990a2d10b668b70a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.918987 23787 tablet_bootstrap.cc:654] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.926029 24670 raft_consensus.cc:493] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea6853efbbcc484eb7529c7e9b37a47a)
I20251023 20:15:28.926183 24670 raft_consensus.cc:515] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.926791 24670 leader_election.cc:290] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.926797 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:28.926913 23333 leader_election.cc:343] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.919428 24363 ts_tablet_manager.cc:1929] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.927628 24363 log.cc:1199] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/925e53d9ea6f4c7baa38551eb917828d
I20251023 20:15:28.926100 24349 ts_tablet_manager.cc:1929] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.928048 24363 ts_tablet_manager.cc:1950] T 925e53d9ea6f4c7baa38551eb917828d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.928098 24349 log.cc:1199] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:28.928584 23333 leader_election.cc:304] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.929162 24821 raft_consensus.cc:2804] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.929240 24821 raft_consensus.cc:493] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.929311 24821 raft_consensus.cc:3060] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.929474 24349 ts_tablet_manager.cc:1950] T d82c92e863b2470699cc822b8cb8c14d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:28.929996 23333 leader_election.cc:343] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 8da82e6d24b14a129b91bb5298139d07
I20251023 20:15:28.928476 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.929852 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:28.931083 23333 leader_election.cc:343] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 8da82e6d24b14a129b91bb5298139d07
I20251023 20:15:28.931174 23333 leader_election.cc:304] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.931641 24670 raft_consensus.cc:2749] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:28.931977 23790 tablet_bootstrap.cc:492] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:28.931962 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.932132 23790 ts_tablet_manager.cc:1403] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20251023 20:15:28.933056 23790 raft_consensus.cc:359] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.933617 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 107) took 3031 ms. Trace:
I20251023 20:15:28.933917 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.934343 24363 rpcz_store.cc:276] 1023 20:15:25.902508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.903184 (+ 676us) service_pool.cc:225] Handling call
1023 20:15:28.898270 (+2995086us) ts_tablet_manager.cc:1151] Deleting tablet 925e53d9ea6f4c7baa38551eb917828d
1023 20:15:28.898282 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.911153 (+ 12871us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.919309 (+ 8156us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.933597 (+ 14288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2994957}
I20251023 20:15:28.936427 23790 raft_consensus.cc:385] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.936605 23790 raft_consensus.cc:740] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.936431 24821 raft_consensus.cc:515] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.936915 24349 tablet_replica.cc:333] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.937348 24821 leader_election.cc:290] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.937716 24349 raft_consensus.cc:2243] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.937875 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48436f3d8f1443db990a2d10b668b70a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:28.938046 23665 raft_consensus.cc:3060] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.937933 24363 tablet_replica.cc:333] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.938207 24349 raft_consensus.cc:2272] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.938392 24363 raft_consensus.cc:2243] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:28.938745 24349 ts_tablet_manager.cc:1916] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.938772 24363 raft_consensus.cc:2272] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.938897 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.939273 24363 ts_tablet_manager.cc:1916] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.940441 23665 raft_consensus.cc:2468] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:28.941226 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.941210 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "48436f3d8f1443db990a2d10b668b70a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:28.941748 23333 leader_election.cc:343] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.941936 23333 leader_election.cc:304] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.942138 24822 raft_consensus.cc:493] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:28.942263 24822 raft_consensus.cc:515] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.942476 23790 consensus_queue.cc:260] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.942324 24821 raft_consensus.cc:2804] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.942828 24822 leader_election.cc:290] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.943341 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.943488 24821 raft_consensus.cc:697] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.943746 24821 consensus_queue.cc:237] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.945472 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.944458 23786 tablet_bootstrap.cc:654] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.946456 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:28.946635 23665 raft_consensus.cc:2468] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:28.947288 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:28.948060 23536 raft_consensus.cc:3060] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.948385 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.950719 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.951552 23790 ts_tablet_manager.cc:1434] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
W20251023 20:15:28.951848 23333 leader_election.cc:343] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.952224 24670 consensus_queue.cc:1048] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:28.953403 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.953528 23787 tablet_bootstrap.cc:492] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:28.953672 23787 ts_tablet_manager.cc:1403] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.005s
I20251023 20:15:28.954061 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.954538 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.954488 23787 raft_consensus.cc:359] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.954649 23787 raft_consensus.cc:385] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:28.954704 23787 raft_consensus.cc:740] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:28.954962 23787 consensus_queue.cc:260] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:28.955207 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.952239 23333 leader_election.cc:304] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:28.955732 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.956528 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.957152 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.957669 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.958256 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.958726 24363 ts_tablet_manager.cc:1929] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.959123 24670 raft_consensus.cc:2804] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:28.959208 24670 raft_consensus.cc:493] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:28.959268 24670 raft_consensus.cc:3060] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.959726 24363 log.cc:1199] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/86b9f4c5d9e0469c87071a33722ce3eb
I20251023 20:15:28.959769 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.958868 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.959307 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.960083 24363 ts_tablet_manager.cc:1950] T 86b9f4c5d9e0469c87071a33722ce3eb P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.960446 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.960804 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 105) took 3058 ms. Trace:
I20251023 20:15:28.960876 24363 rpcz_store.cc:276] 1023 20:15:25.902350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.903330 (+ 980us) service_pool.cc:225] Handling call
1023 20:15:28.937881 (+3034551us) ts_tablet_manager.cc:1151] Deleting tablet 86b9f4c5d9e0469c87071a33722ce3eb
1023 20:15:28.937891 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.946492 (+ 8601us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.958612 (+ 12120us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.960787 (+ 2175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3034434}
I20251023 20:15:28.961001 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:28.961637 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.961762 24670 raft_consensus.cc:515] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.962319 24670 leader_election.cc:290] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:28.962422 23787 ts_tablet_manager.cc:1434] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251023 20:15:28.962533 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.963114 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.963970 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.965193 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.965780 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.966377 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.966951 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.967456 24349 ts_tablet_manager.cc:1929] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.968289 24363 tablet_replica.cc:333] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.968611 24363 raft_consensus.cc:2243] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.968864 24363 raft_consensus.cc:2272] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.969708 24363 ts_tablet_manager.cc:1916] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.970073 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:28.972147 23665 raft_consensus.cc:3060] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.974702 23665 raft_consensus.cc:2468] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:28.974927 23787 tablet_bootstrap.cc:492] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:28.967502 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.967965 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.975118 24349 log.cc:1199] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:28.968977 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.970057 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:28.975442 24349 ts_tablet_manager.cc:1950] T cec51c54ea534b1da4b1d170e79d9f1c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:28.968427 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:28.976250 23333 leader_election.cc:304] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:28.976972 23790 tablet_bootstrap.cc:492] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:28.977361 24825 raft_consensus.cc:2804] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:28.977494 24825 raft_consensus.cc:697] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:28.977748 24825 consensus_queue.cc:237] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:28.978987 23790 tablet_bootstrap.cc:654] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.980144 24349 tablet_replica.cc:333] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:28.980412 24349 raft_consensus.cc:2243] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.980647 24349 raft_consensus.cc:2272] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.981508 24349 ts_tablet_manager.cc:1916] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.982296 24363 ts_tablet_manager.cc:1929] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:28.982445 24363 log.cc:1199] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/c97e2e0f839a4ef389fe100b6a81301e
I20251023 20:15:28.982733 24363 ts_tablet_manager.cc:1950] T c97e2e0f839a4ef389fe100b6a81301e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:28.983439 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 111) took 3079 ms. Trace:
I20251023 20:15:28.983541 24363 rpcz_store.cc:276] 1023 20:15:25.903743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.903818 (+ 75us) service_pool.cc:225] Handling call
1023 20:15:28.968233 (+3064415us) ts_tablet_manager.cc:1151] Deleting tablet c97e2e0f839a4ef389fe100b6a81301e
1023 20:15:28.968245 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.979351 (+ 11106us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.982179 (+ 2828us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:28.983377 (+ 1198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3064265}
I20251023 20:15:28.983965 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 110) took 3080 ms. Trace:
I20251023 20:15:28.984054 24363 rpcz_store.cc:276] 1023 20:15:25.903652 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:25.904079 (+ 427us) service_pool.cc:225] Handling call
1023 20:15:28.983819 (+3079740us) ts_tablet_manager.cc:1151] Deleting tablet cbacaf2fea24405caaf84d59b625fca3
1023 20:15:28.983829 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.983918 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3079604}
I20251023 20:15:28.984404 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 113) took 2531 ms. Trace:
I20251023 20:15:28.984490 24363 rpcz_store.cc:276] 1023 20:15:26.452526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.459661 (+ 7135us) service_pool.cc:225] Handling call
1023 20:15:28.984277 (+2524616us) ts_tablet_manager.cc:1151] Deleting tablet 1e19859a65a144598e1612be89daa3a1
1023 20:15:28.984286 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:28.984358 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2524371}
I20251023 20:15:28.984779 24363 tablet_replica.cc:333] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:28.984975 24363 raft_consensus.cc:2243] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:28.985687 23529 raft_consensus.cc:3060] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:28.989897 24780 ts_tablet_manager.cc:1929] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:28.991663 24780 log.cc:1199] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:28.992205 23786 tablet_bootstrap.cc:492] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
W20251023 20:15:28.994880 23333 leader_election.cc:343] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:28.994841 23786 ts_tablet_manager.cc:1403] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.000s
I20251023 20:15:28.997210 24363 raft_consensus.cc:2272] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:28.997773 23787 tablet_bootstrap.cc:654] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:28.997897 24363 ts_tablet_manager.cc:1916] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:28.999987 24780 ts_tablet_manager.cc:1950] T cb915d114fc0421999d079c23cad8eb5 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.000336 24349 ts_tablet_manager.cc:1929] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.000442 24349 log.cc:1199] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/c97e2e0f839a4ef389fe100b6a81301e
I20251023 20:15:29.000797 24349 ts_tablet_manager.cc:1950] T c97e2e0f839a4ef389fe100b6a81301e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.004058 23786 raft_consensus.cc:359] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.004273 23786 raft_consensus.cc:385] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.004350 23786 raft_consensus.cc:740] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.004551 24349 tablet_replica.cc:333] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.004791 24349 raft_consensus.cc:2243] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.005064 23790 tablet_bootstrap.cc:492] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.005249 23790 ts_tablet_manager.cc:1403] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20251023 20:15:29.005383 24363 ts_tablet_manager.cc:1929] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:29.005570 24363 log.cc:1199] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e5368813164949ff80e27bf59dcc931a
I20251023 20:15:29.005936 24363 ts_tablet_manager.cc:1950] T e5368813164949ff80e27bf59dcc931a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.006075 23790 raft_consensus.cc:359] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.006263 23790 raft_consensus.cc:385] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.006345 23790 raft_consensus.cc:740] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.006600 23790 consensus_queue.cc:260] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.006723 24349 raft_consensus.cc:2272] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.007365 23790 ts_tablet_manager.cc:1434] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:29.007705 23790 tablet_bootstrap.cc:492] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:29.007745 23333 consensus_peers.cc:597] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2532c8567db14901b383f42e5f04b4c2. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.009044 24349 ts_tablet_manager.cc:1916] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.009339 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.008338 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 115) took 2442 ms. Trace:
I20251023 20:15:29.009667 24363 rpcz_store.cc:276] 1023 20:15:26.566081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:26.566136 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:28.984698 (+2418562us) ts_tablet_manager.cc:1151] Deleting tablet e5368813164949ff80e27bf59dcc931a
1023 20:15:28.984706 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.001211 (+ 16505us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.005250 (+ 4039us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.008301 (+ 3051us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2418360}
I20251023 20:15:29.010006 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.010540 24363 tablet_replica.cc:333] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.010581 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.010756 24363 raft_consensus.cc:2243] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.010908 24363 raft_consensus.cc:2272] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.011240 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.011440 24363 ts_tablet_manager.cc:1916] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.011816 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.014261 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.014817 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet cb915d114fc0421999d079c23cad8eb5 (table default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]) successfully deleted
I20251023 20:15:29.015213 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.015354 23786 consensus_queue.cc:260] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.015882 23786 ts_tablet_manager.cc:1434] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20251023 20:15:29.016860 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:29.017158 23205 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because the tablet was not found. No further retry: Not found: Tablet not found: cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:29.019260 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:29.019622 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
I20251023 20:15:29.020269 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.020443 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.020592 23786 tablet_bootstrap.cc:492] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.020663 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.020999 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.021188 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.021344 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.021486 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.021793 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.022034 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.022337 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.022604 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet e5368813164949ff80e27bf59dcc931a with delete_type TABLET_DATA_DELETED (replaced by tablet abdad7d76eb8441496d38642382f74bb) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.023494 23786 tablet_bootstrap.cc:654] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.023168 23790 tablet_bootstrap.cc:654] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:29.026083 23203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20251023 20:15:29.024137 24780 tablet_replica.cc:333] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
W20251023 20:15:29.027071 23203 consensus_peers.cc:597] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a -> Peer ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509): Couldn't send request to peer ae5a310b4a514c0f812bda133c756062. Status: Network error: Client connection negotiation failed: client connection to 127.22.127.189:33509: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251023 20:15:29.025579 24821 raft_consensus.cc:493] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.027410 24821 raft_consensus.cc:515] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.028069 24821 leader_election.cc:290] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.029888 24821 raft_consensus.cc:493] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.030072 24821 raft_consensus.cc:515] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.030761 24821 leader_election.cc:290] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.032603 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.032652 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.032856 23665 raft_consensus.cc:2468] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:29.033358 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f103bece2fa240c19c23ac51046c13ed" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.033514 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f103bece2fa240c19c23ac51046c13ed" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.033669 23665 raft_consensus.cc:2468] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:29.033924 23333 leader_election.cc:343] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:29.034101 23333 leader_election.cc:343] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.034276 23333 leader_election.cc:304] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.034497 24821 raft_consensus.cc:2804] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.034626 24821 raft_consensus.cc:493] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.034745 24821 raft_consensus.cc:3060] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.036229 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet aa794b3bec4b4b3993c814ea7cd02ee2 (table default.Table_083a66aed4484a38961dd79edd4d1c2e [id=a431259b5e254da6941241ffcf8ce597]) successfully deleted
W20251023 20:15:29.038542 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a86a25659ec44d7c825783036342ed45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a86a25659ec44d7c825783036342ed45 already in progress: creating tablet
W20251023 20:15:29.040714 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6558265ff9e249c68123577958f1401a because the tablet was not found. No further retry: Not found: Tablet not found: 6558265ff9e249c68123577958f1401a
I20251023 20:15:29.043707 24780 raft_consensus.cc:2243] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.044050 24780 raft_consensus.cc:2272] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.044406 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a8eaa2167f82475ca2f6a110aba288ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8eaa2167f82475ca2f6a110aba288ec already in progress: creating tablet
I20251023 20:15:29.044613 23787 tablet_bootstrap.cc:492] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.044636 24780 ts_tablet_manager.cc:1916] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.044896 23787 ts_tablet_manager.cc:1403] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.004s
W20251023 20:15:29.045552 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 352812b56f8d4feb818a364d6adfd86d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352812b56f8d4feb818a364d6adfd86d already in progress: creating tablet
I20251023 20:15:29.034634 23333 leader_election.cc:304] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.045713 23787 raft_consensus.cc:359] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.046092 24825 raft_consensus.cc:2804] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.046167 24825 raft_consensus.cc:493] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.046181 23787 raft_consensus.cc:385] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.046231 24825 raft_consensus.cc:3060] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:29.046213 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6a54470d088748b0be22ab9db5a25f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a54470d088748b0be22ab9db5a25f87 already in progress: creating tablet
I20251023 20:15:29.046272 23787 raft_consensus.cc:740] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.046440 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d82c92e863b2470699cc822b8cb8c14d (table default.Table_8ac87d9615a74c5c858be1d6c31772a6 [id=1740292e42a544e5bd5d834a281bddbc]) successfully deleted
I20251023 20:15:29.046516 23787 consensus_queue.cc:260] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.046638 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
I20251023 20:15:29.048343 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet cec51c54ea534b1da4b1d170e79d9f1c (table default.Table_21f0d4fca62a4309816edd74c6b137bf [id=be3ef152e4fd4281b5caf942c8d404ab]) successfully deleted
W20251023 20:15:29.048686 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 600f0110f8bd4b05983d5922d871c266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600f0110f8bd4b05983d5922d871c266 already in progress: creating tablet
I20251023 20:15:29.049268 24825 raft_consensus.cc:515] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.050210 24825 leader_election.cc:290] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.051831 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f103bece2fa240c19c23ac51046c13ed" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.051975 24297 raft_consensus.cc:493] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.052090 24297 raft_consensus.cc:515] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.052321 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f103bece2fa240c19c23ac51046c13ed" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.052482 23664 raft_consensus.cc:3060] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:29.052387 23333 leader_election.cc:343] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.052585 24297 leader_election.cc:290] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.053284 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a636a4cdeb404e9f806d8e166ea57938" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.053596 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ef1229b9e0384a82811f244d2bf4c8ce because the tablet was not found. No further retry: Not found: Tablet not found: ef1229b9e0384a82811f244d2bf4c8ce
W20251023 20:15:29.054181 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a636a4cdeb404e9f806d8e166ea57938 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a636a4cdeb404e9f806d8e166ea57938 already in progress: creating tablet
I20251023 20:15:29.054994 23664 raft_consensus.cc:2468] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.055941 23787 ts_tablet_manager.cc:1434] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
W20251023 20:15:29.056023 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f103bece2fa240c19c23ac51046c13ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f103bece2fa240c19c23ac51046c13ed already in progress: creating tablet
I20251023 20:15:29.056051 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a636a4cdeb404e9f806d8e166ea57938" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.056280 23664 raft_consensus.cc:2468] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:29.056829 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9f422575c7c4251879130501ab721fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f422575c7c4251879130501ab721fd already in progress: creating tablet
I20251023 20:15:29.057031 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet c97e2e0f839a4ef389fe100b6a81301e (table default.Table_5d14af3d31254be5a7ff6cb5d1fea162 [id=55a8a0300c454094a6ee9c2c7897495c]) successfully deleted
I20251023 20:15:29.057147 23212 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet d2ae19cc80194363b8a9c96fbafb4ebe
I20251023 20:15:29.058626 23212 catalog_manager.cc:6719] ReplaceTablet: tablet d2ae19cc80194363b8a9c96fbafb4ebe deleted and replaced by tablet 94631fca1b834bc89791cb07fdc93523
I20251023 20:15:29.058847 23212 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 58) took 3345 ms. Trace:
I20251023 20:15:29.059001 23212 rpcz_store.cc:276] 1023 20:15:25.713599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.858954 (+3145355us) service_pool.cc:225] Handling call
1023 20:15:29.058788 (+199834us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:28.897878 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.897990 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.897997 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.897999 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:28.898106 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.898116 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.898120 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:28.898190 (+ 70us) tablet.cc:699] Row locks acquired
1023 20:15:28.898193 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:28.898228 (+ 35us) write_op.cc:270] Start()
1023 20:15:28.898260 (+ 32us) write_op.cc:276] Timestamp: P: 1761250528898189 usec, L: 0
1023 20:15:28.898264 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.898431 (+ 167us) log.cc:844] Serialized 665 byte log entry
1023 20:15:28.908335 (+ 9904us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.908649 (+ 314us) write_op.cc:301] APPLY: starting
1023 20:15:28.908664 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.908702 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.908733 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.908847 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.908898 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.908905 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:28.908974 (+ 69us) log.cc:844] Serialized 29 byte log entry
1023 20:15:28.909030 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.909046 (+ 16us) write_op.cc:454] Released schema lock
1023 20:15:28.909052 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.909138 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.909142 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":260,"apply.run_cpu_time_us":484,"apply.run_wall_time_us":483,"mutex_wait_us":37102,"num_ops":2,"prepare.queue_time_us":37864,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":644,"raft.queue_time_us":345,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":10042}]]}
I20251023 20:15:29.060048 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:29.060293 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:29.060881 23787 tablet_bootstrap.cc:492] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.060554 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:29.061368 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:29.061892 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:29.062214 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:29.062511 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:29.062729 23787 tablet_bootstrap.cc:654] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.063099 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:29.063884 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:29.064262 23216 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 59) took 3344 ms. Trace:
I20251023 20:15:29.064419 23216 rpcz_store.cc:276] 1023 20:15:25.720002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.860865 (+3140863us) service_pool.cc:225] Handling call
1023 20:15:28.861928 (+ 1063us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:28.863380 (+ 1452us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:28.917574 (+ 54194us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:29.064109 (+146535us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:29.064204 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:28.898952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.899044 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.899049 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:28.899051 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:28.899129 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.899137 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.899142 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:28.899206 (+ 64us) tablet.cc:699] Row locks acquired
1023 20:15:28.899209 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:28.899239 (+ 30us) write_op.cc:270] Start()
1023 20:15:28.899265 (+ 26us) write_op.cc:276] Timestamp: P: 1761250528899201 usec, L: 0
1023 20:15:28.899268 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.899416 (+ 148us) log.cc:844] Serialized 433 byte log entry
1023 20:15:28.908400 (+ 8984us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.909210 (+ 810us) write_op.cc:301] APPLY: starting
1023 20:15:28.909221 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.909253 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.909285 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.909369 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.909395 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:28.909401 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:28.909457 (+ 56us) log.cc:844] Serialized 29 byte log entry
1023 20:15:28.909492 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.909507 (+ 15us) write_op.cc:454] Released schema lock
1023 20:15:28.909512 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":779,"apply.run_cpu_time_us":353,"apply.run_wall_time_us":353,"mutex_wait_us":25367,"num_ops":2,"prepare.queue_time_us":25406,"prepare.run_cpu_time_us":544,"prepare.run_wall_time_us":543,"raft.queue_time_us":51,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":3,"replication_time_us":9116,"thread_start_us":44059,"threads_started":1}]]}
W20251023 20:15:29.065301 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cbacaf2fea24405caaf84d59b625fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbacaf2fea24405caaf84d59b625fca3 already in progress: creating tablet
I20251023 20:15:29.066394 24821 raft_consensus.cc:515] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.067062 24821 leader_election.cc:290] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.065553 23216 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_03569553d00b42c890bd05b75ff28289"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:29.069828 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 83dae34129794f49b43038d3b1636ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83dae34129794f49b43038d3b1636ea0 already in progress: creating tablet
I20251023 20:15:29.065871 24363 ts_tablet_manager.cc:1929] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.071261 24363 log.cc:1199] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/af899c01b4af46d39d27f984cc3d5940
I20251023 20:15:29.071630 24363 ts_tablet_manager.cc:1950] T af899c01b4af46d39d27f984cc3d5940 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.072247 24349 ts_tablet_manager.cc:1929] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.072423 24349 log.cc:1199] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a9c5b69df4c04365b1ac1457fe8300cc
I20251023 20:15:29.072723 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 131) took 1705 ms. Trace:
I20251023 20:15:29.072883 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.073130 23664 raft_consensus.cc:3060] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.065816 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:29.063434 23215 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 57) took 3391 ms. Trace:
W20251023 20:15:29.073336 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
I20251023 20:15:29.073347 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe with delete_type TABLET_DATA_DELETED (replaced by tablet 94631fca1b834bc89791cb07fdc93523) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:29.073536 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 1a64a7c2e1f242e1b54243489622bae5 (table default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0 [id=643c30acd68743bfb7c0ebd7b98199e7]) successfully deleted
I20251023 20:15:29.072851 24363 rpcz_store.cc:276] 1023 20:15:27.367239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:27.370511 (+ 3272us) service_pool.cc:225] Handling call
1023 20:15:29.010466 (+1639955us) ts_tablet_manager.cc:1151] Deleting tablet af899c01b4af46d39d27f984cc3d5940
1023 20:15:29.010476 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.043940 (+ 33464us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.065753 (+ 21813us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.072666 (+ 6913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639728}
W20251023 20:15:29.073755 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet bb475e303d07491bbc19dc3f628bceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb475e303d07491bbc19dc3f628bceb7 already in progress: creating tablet
I20251023 20:15:29.073935 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 140) took 1064 ms. Trace:
I20251023 20:15:29.073930 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 1ae910ac41fd4e349f159ab27e44375a (table default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]) successfully deleted
I20251023 20:15:29.073446 23215 rpcz_store.cc:276] 1023 20:15:25.671512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.854745 (+3183233us) service_pool.cc:225] Handling call
1023 20:15:28.857270 (+ 2525us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:28.857735 (+ 465us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:28.919355 (+ 61620us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:29.063282 (+143927us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:29.063392 (+ 110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:28.896335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.896461 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.896468 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.896470 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:28.896570 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.896585 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.896590 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:28.896671 (+ 81us) tablet.cc:699] Row locks acquired
1023 20:15:28.896673 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.896709 (+ 36us) write_op.cc:270] Start()
1023 20:15:28.896744 (+ 35us) write_op.cc:276] Timestamp: P: 1761250528896671 usec, L: 0
1023 20:15:28.896748 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.896944 (+ 196us) log.cc:844] Serialized 433 byte log entry
1023 20:15:28.904719 (+ 7775us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.904880 (+ 161us) write_op.cc:301] APPLY: starting
1023 20:15:28.904897 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.904942 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.904966 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.905071 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.905128 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:28.905136 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:28.905219 (+ 83us) log.cc:844] Serialized 29 byte log entry
1023 20:15:28.905282 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.905301 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:28.905307 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.905378 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.905382 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":494,"apply.run_wall_time_us":493,"num_ops":2,"prepare.queue_time_us":38058,"prepare.run_cpu_time_us":728,"prepare.run_wall_time_us":728,"raft.queue_time_us":1010,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":258,"replication_time_us":7939}]]}
I20251023 20:15:29.074018 24363 rpcz_store.cc:276] 1023 20:15:28.008957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.009131 (+ 174us) service_pool.cc:225] Handling call
1023 20:15:29.073781 (+1064650us) ts_tablet_manager.cc:1151] Deleting tablet 9a4f012469fa4e8e827e485287ad76dd
1023 20:15:29.073791 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.073878 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064489}
I20251023 20:15:29.074235 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 925e53d9ea6f4c7baa38551eb917828d (table default.Table_da2bae6bea6047318cd9fb653b35475c [id=275f37e241644f14bba90f90529ebdce]) successfully deleted
I20251023 20:15:29.074399 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 86b9f4c5d9e0469c87071a33722ce3eb (table default.Table_b3b5199d1faf4e5795ba391229812ea9 [id=8ec2443a50674a67a3393236ee4bc48e]) successfully deleted
W20251023 20:15:29.072089 23216 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03569553d00b42c890bd05b75ff28289 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:29.073360 23333 consensus_peers.cc:597] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d951134f6773410386bc57f7f90310c3. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:29.075258 23333 consensus_peers.cc:597] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a64a7c2e1f242e1b54243489622bae5. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.075381 23333 leader_election.cc:304] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.075693 23664 raft_consensus.cc:2468] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.075754 23333 leader_election.cc:304] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:29.076148 23333 leader_election.cc:343] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.076161 24670 raft_consensus.cc:2804] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.076370 24821 raft_consensus.cc:2804] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.076427 24821 raft_consensus.cc:493] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.076452 24670 raft_consensus.cc:697] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.076480 24821 raft_consensus.cc:3060] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.076716 24670 consensus_queue.cc:237] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.079020 24821 raft_consensus.cc:515] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.083532 24363 tablet_replica.cc:333] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.083993 24363 raft_consensus.cc:2243] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.072911 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.082743 23529 raft_consensus.cc:3060] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.084653 24363 raft_consensus.cc:2272] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.085943 23786 tablet_bootstrap.cc:492] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.086063 23786 ts_tablet_manager.cc:1403] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.002s
I20251023 20:15:29.086316 23215 master_service.cc:954] ReplaceTablet: received request to replace tablet 514cb3b1ed024041ac160a7d9acff618 from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:29.086393 24363 ts_tablet_manager.cc:1916] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.086769 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:29.086877 23786 raft_consensus.cc:359] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.086997 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:29.087019 23786 raft_consensus.cc:385] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.087066 23786 raft_consensus.cc:740] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.087159 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:29.087373 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:29.087703 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:29.082111 23790 tablet_bootstrap.cc:492] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.088444 23786 consensus_queue.cc:260] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.088688 23790 ts_tablet_manager.cc:1403] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.003s
I20251023 20:15:29.089051 23787 tablet_bootstrap.cc:492] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.089532 23787 ts_tablet_manager.cc:1403] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20251023 20:15:29.089481 23790 raft_consensus.cc:359] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.089731 23790 raft_consensus.cc:385] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.089864 23790 raft_consensus.cc:740] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.090221 23790 consensus_queue.cc:260] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.090312 23787 raft_consensus.cc:359] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.090572 23787 raft_consensus.cc:385] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.090826 23786 ts_tablet_manager.cc:1434] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:29.090679 23787 raft_consensus.cc:740] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.091387 23786 tablet_bootstrap.cc:492] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.091531 23790 ts_tablet_manager.cc:1434] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:29.091202 23787 consensus_queue.cc:260] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.092423 23790 tablet_bootstrap.cc:492] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.092773 23786 tablet_bootstrap.cc:654] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.083716 23333 leader_election.cc:304] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:29.093803 24822 raft_consensus.cc:2804] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.094156 24822 raft_consensus.cc:697] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.094384 24822 consensus_queue.cc:237] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.095767 24349 ts_tablet_manager.cc:1950] T a9c5b69df4c04365b1ac1457fe8300cc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:29.095291 23333 leader_election.cc:343] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.096436 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:29.096920 23213 catalog_manager.cc:5649] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:29.097396 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:29.097645 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:29.098289 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:29.098309 23787 ts_tablet_manager.cc:1434] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20251023 20:15:29.098546 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:29.098811 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:29.099316 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:29.099439 24349 tablet_replica.cc:333] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.099651 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:29.100042 24349 raft_consensus.cc:2243] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.100225 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:29.100530 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:29.100734 24349 raft_consensus.cc:2272] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.101423 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:29.101686 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:29.102195 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:29.102851 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:29.103579 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:29.103492 23216 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 60) took 3369 ms. Trace:
I20251023 20:15:29.103953 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:29.103794 23216 rpcz_store.cc:276] 1023 20:15:25.733980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.065279 (+3331299us) service_pool.cc:225] Handling call
1023 20:15:29.096142 (+ 30863us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:29.096644 (+ 502us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:29.103170 (+ 6526us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.103175 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:29.103247 (+ 72us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:29.103347 (+ 100us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:29.103436 (+ 89us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:29.097259 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:29.097375 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:29.097385 (+ 10us) write_op.cc:435] Acquired schema lock
1023 20:15:29.097388 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:29.097484 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:29.097497 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:29.097502 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:29.097646 (+ 144us) tablet.cc:699] Row locks acquired
1023 20:15:29.097648 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:29.097685 (+ 37us) write_op.cc:270] Start()
1023 20:15:29.097720 (+ 35us) write_op.cc:276] Timestamp: P: 1761250529097663 usec, L: 0
1023 20:15:29.097723 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:29.097893 (+ 170us) log.cc:844] Serialized 433 byte log entry
1023 20:15:29.101663 (+ 3770us) op_driver.cc:464] REPLICATION: finished
1023 20:15:29.101812 (+ 149us) write_op.cc:301] APPLY: starting
1023 20:15:29.101831 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:29.101890 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:29.101953 (+ 63us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:29.102120 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:29.102156 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:29.102164 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:29.102273 (+ 109us) log.cc:844] Serialized 29 byte log entry
1023 20:15:29.102337 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.102359 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:29.102365 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"apply.run_cpu_time_us":625,"apply.run_wall_time_us":625,"num_ops":2,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":749,"raft.queue_time_us":885,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":284,"replication_time_us":3910}]]}
I20251023 20:15:29.104612 23787 tablet_bootstrap.cc:492] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.104753 24349 ts_tablet_manager.cc:1916] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.104960 23216 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_b01089ea89ef4536a351098a4caa38f0" } new_table_name: "default.Table_5ba404e4eee84ed4bf30bf1be1236aa5" modify_external_catalogs: true
I20251023 20:15:29.107086 24423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251023 20:15:29.108636 23787 tablet_bootstrap.cc:654] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.110320 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet c97e2e0f839a4ef389fe100b6a81301e (table default.Table_5d14af3d31254be5a7ff6cb5d1fea162 [id=55a8a0300c454094a6ee9c2c7897495c]) successfully deleted
W20251023 20:15:29.110600 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cbacaf2fea24405caaf84d59b625fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbacaf2fea24405caaf84d59b625fca3 already in progress: creating tablet
I20251023 20:15:29.111044 24349 ts_tablet_manager.cc:1929] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.111222 24349 log.cc:1199] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/1a64a7c2e1f242e1b54243489622bae5
I20251023 20:15:29.111598 24349 ts_tablet_manager.cc:1950] T 1a64a7c2e1f242e1b54243489622bae5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.113526 24349 tablet_replica.cc:333] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.113751 24349 raft_consensus.cc:2243] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.113953 24349 raft_consensus.cc:2272] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.114333 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:29.114544 24349 ts_tablet_manager.cc:1916] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.114634 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:29.114809 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:29.114979 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:29.115283 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:29.116209 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:29.117292 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:29.117935 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:29.118369 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:29.118542 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:29.118691 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:29.118934 23213 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:29.119982 24423 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20251023 20:15:29.120121 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e19859a65a144598e1612be89daa3a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e19859a65a144598e1612be89daa3a1 already in progress: creating tablet
I20251023 20:15:29.120303 24423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251023 20:15:29.120376 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet e5368813164949ff80e27bf59dcc931a (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]) successfully deleted
I20251023 20:15:29.120585 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]) successfully deleted
W20251023 20:15:29.120784 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9a4f012469fa4e8e827e485287ad76dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a4f012469fa4e8e827e485287ad76dd already in progress: creating tablet
W20251023 20:15:29.121009 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d62dbf044acd4459b26d2e7e50f6206e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d62dbf044acd4459b26d2e7e50f6206e already in progress: creating tablet
W20251023 20:15:29.121224 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 377de4ea0bd6489284fac6b0fab52c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377de4ea0bd6489284fac6b0fab52c8c already in progress: creating tablet
I20251023 20:15:29.121261 24349 ts_tablet_manager.cc:1929] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.121438 24349 log.cc:1199] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/925e53d9ea6f4c7baa38551eb917828d
W20251023 20:15:29.121443 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
W20251023 20:15:29.121626 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
W20251023 20:15:29.121841 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 95574277dde44983bb0a43d2b5ad01d5 because the tablet was not found. No further retry: Not found: Tablet not found: 95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:29.121888 24349 ts_tablet_manager.cc:1950] T 925e53d9ea6f4c7baa38551eb917828d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:29.122036 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
W20251023 20:15:29.122224 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:29.122412 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet aa794b3bec4b4b3993c814ea7cd02ee2 because the tablet was not found. No further retry: Not found: Tablet not found: aa794b3bec4b4b3993c814ea7cd02ee2
I20251023 20:15:29.122546 24821 leader_election.cc:290] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:29.122630 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
I20251023 20:15:29.123275 24349 tablet_replica.cc:333] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.123478 24349 raft_consensus.cc:2243] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.123643 24349 raft_consensus.cc:2272] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.123555 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a636a4cdeb404e9f806d8e166ea57938" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.123775 23664 raft_consensus.cc:3060] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.124487 24349 ts_tablet_manager.cc:1916] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.126374 23664 raft_consensus.cc:2468] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:29.129776 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
I20251023 20:15:29.130196 24349 ts_tablet_manager.cc:1929] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.130390 24349 log.cc:1199] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a708cd011802458aa0e2991fd4b4a2d3
I20251023 20:15:29.130563 23786 tablet_bootstrap.cc:492] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.130694 23786 ts_tablet_manager.cc:1403] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20251023 20:15:29.130867 24349 ts_tablet_manager.cc:1950] T a708cd011802458aa0e2991fd4b4a2d3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.131196 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a636a4cdeb404e9f806d8e166ea57938" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.131425 23786 raft_consensus.cc:359] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:29.131588 23786 raft_consensus.cc:385] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.131644 23786 raft_consensus.cc:740] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:29.131798 23333 leader_election.cc:343] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.131879 23786 consensus_queue.cc:260] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:29.132287 24349 tablet_replica.cc:333] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.132375 23786 ts_tablet_manager.cc:1434] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:29.132520 24349 raft_consensus.cc:2243] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:29.133332 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
I20251023 20:15:29.131997 23333 leader_election.cc:304] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.133072 23786 tablet_bootstrap.cc:492] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.133838 24821 raft_consensus.cc:2804] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.133971 24821 raft_consensus.cc:697] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.134902 24821 consensus_queue.cc:237] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.135246 24780 ts_tablet_manager.cc:1929] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.135351 24780 log.cc:1199] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:29.135360 24349 raft_consensus.cc:2272] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.135514 24658 raft_consensus.cc:493] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.135641 24658 raft_consensus.cc:515] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.135823 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe with delete_type TABLET_DATA_DELETED (replaced by tablet 94631fca1b834bc89791cb07fdc93523) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:29.135869 24349 ts_tablet_manager.cc:1916] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.136257 24658 leader_election.cc:290] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.136405 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe with delete_type TABLET_DATA_DELETED (replaced by tablet 94631fca1b834bc89791cb07fdc93523) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:29.136919 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a9c5b69df4c04365b1ac1457fe8300cc (table default.Table_8c497d9a1dcc41aa8466741225b4de85 [id=5746055cb51b4ee4b54b672987471cb1]) successfully deleted
I20251023 20:15:29.137095 23786 tablet_bootstrap.cc:654] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.137226 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 1a64a7c2e1f242e1b54243489622bae5 (table default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0 [id=643c30acd68743bfb7c0ebd7b98199e7]) successfully deleted
W20251023 20:15:29.137733 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 8da82e6d24b14a129b91bb5298139d07 because the tablet was not found. No further retry: Not found: Tablet not found: 8da82e6d24b14a129b91bb5298139d07
W20251023 20:15:29.138005 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet af899c01b4af46d39d27f984cc3d5940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af899c01b4af46d39d27f984cc3d5940 already in progress: creating tablet
W20251023 20:15:29.138250 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 523d72012c364457b9a2a299f2f789a1 because the tablet was not found. No further retry: Not found: Tablet not found: 523d72012c364457b9a2a299f2f789a1
I20251023 20:15:29.138285 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5cb6b4e2e5e4a2f9183111cfd54ec90" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.138504 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet bb475e303d07491bbc19dc3f628bceb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb475e303d07491bbc19dc3f628bceb7 already in progress: creating tablet
I20251023 20:15:29.138751 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 925e53d9ea6f4c7baa38551eb917828d (table default.Table_da2bae6bea6047318cd9fb653b35475c [id=275f37e241644f14bba90f90529ebdce]) successfully deleted
W20251023 20:15:29.138777 23604 leader_election.cc:343] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.138973 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a708cd011802458aa0e2991fd4b4a2d3 (table default.Table_d2b87a7247624b0380ee21ec002b9f77 [id=f64a3298a8d44a7ba1700cac9bf91080]) successfully deleted
I20251023 20:15:29.140879 24363 ts_tablet_manager.cc:1929] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.143940 24363 log.cc:1199] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:29.144922 24363 ts_tablet_manager.cc:1950] T cb915d114fc0421999d079c23cad8eb5 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.145659 24349 ts_tablet_manager.cc:1929] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.146463 24349 log.cc:1199] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/4b6c3e7169154e3497dd3570dca0d745
W20251023 20:15:29.146747 23333 consensus_peers.cc:597] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2532c8567db14901b383f42e5f04b4c2. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.147274 24825 raft_consensus.cc:493] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.147409 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5cb6b4e2e5e4a2f9183111cfd54ec90" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.147389 24825 raft_consensus.cc:515] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.147931 24825 leader_election.cc:290] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:29.147979 23605 leader_election.cc:343] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.148115 23605 leader_election.cc:304] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.147120 23786 tablet_bootstrap.cc:492] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.148485 24658 raft_consensus.cc:2749] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.148552 23786 ts_tablet_manager.cc:1403] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251023 20:15:29.146991 23787 tablet_bootstrap.cc:492] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.149276 23787 ts_tablet_manager.cc:1403] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.006s
I20251023 20:15:29.149438 23786 raft_consensus.cc:359] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.149592 23786 raft_consensus.cc:385] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.149561 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet cb915d114fc0421999d079c23cad8eb5 (table default.Table_d880d87e1b2244b8ba05b6ff6653e35b [id=936e505ef2f7447db8f9cb85af99c6d8]) successfully deleted
I20251023 20:15:29.149653 23786 raft_consensus.cc:740] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.149818 23790 tablet_bootstrap.cc:654] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.149879 23786 consensus_queue.cc:260] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.150305 23786 ts_tablet_manager.cc:1434] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:29.151254 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.151798 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.151997 23786 tablet_bootstrap.cc:492] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.151993 23664 raft_consensus.cc:2468] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:29.153126 24780 ts_tablet_manager.cc:1950] T 95574277dde44983bb0a43d2b5ad01d5 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.154613 24780 tablet_replica.cc:333] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.154682 23333 leader_election.cc:304] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:29.154805 24780 raft_consensus.cc:2243] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.155364 23786 tablet_bootstrap.cc:654] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.156272 24349 ts_tablet_manager.cc:1950] T 4b6c3e7169154e3497dd3570dca0d745 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.156441 24825 raft_consensus.cc:2804] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.156481 24780 raft_consensus.cc:2272] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.156509 24825 raft_consensus.cc:493] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.156566 24825 raft_consensus.cc:3060] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.156952 24780 ts_tablet_manager.cc:1916] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.159214 24825 raft_consensus.cc:515] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.159240 23787 raft_consensus.cc:359] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.159402 23787 raft_consensus.cc:385] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.159514 23787 raft_consensus.cc:740] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.159775 23787 consensus_queue.cc:260] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.160006 24825 leader_election.cc:290] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:29.160291 23333 leader_election.cc:343] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.160331 23787 ts_tablet_manager.cc:1434] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251023 20:15:29.162190 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 95574277dde44983bb0a43d2b5ad01d5 (table default.Table_2d1938985e9345eba1d4fb750b552e78 [id=8b537204d252421c8ef4084d1dd9226f]) successfully deleted
W20251023 20:15:29.162473 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a86a25659ec44d7c825783036342ed45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a86a25659ec44d7c825783036342ed45 already in progress: creating tablet
I20251023 20:15:29.162633 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.164166 24349 tablet_replica.cc:333] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.164381 24349 raft_consensus.cc:2243] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:29.164649 23333 leader_election.cc:343] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.164892 24349 raft_consensus.cc:2272] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.165455 24349 ts_tablet_manager.cc:1916] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:29.165756 23333 consensus_peers.cc:597] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a21ec49194e24723b31fd9c86f1f85e4. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.170791 23215 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 61) took 3250 ms. Trace:
I20251023 20:15:29.170898 23215 rpcz_store.cc:276] 1023 20:15:25.919884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.074314 (+3154430us) service_pool.cc:225] Handling call
1023 20:15:29.170730 (+ 96416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:29.171399 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_c462519a21af4c17bc119f202f50922a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:29.172010 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c462519a21af4c17bc119f202f50922a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:29.174571 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 4b6c3e7169154e3497dd3570dca0d745 (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]) successfully deleted
I20251023 20:15:29.177032 24349 ts_tablet_manager.cc:1929] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.177525 24349 log.cc:1199] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:29.177999 24349 ts_tablet_manager.cc:1950] T 951b2276d3324c2d999c8d7fb97767bc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.179457 24822 raft_consensus.cc:493] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:29.179625 24822 raft_consensus.cc:515] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.180996 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 951b2276d3324c2d999c8d7fb97767bc (table default.Table_441bf8050be9453c85aef2cb7ac63ea4 [id=425e2178647a42aaa99cef01f06523f1]) successfully deleted
I20251023 20:15:29.181483 24822 leader_election.cc:290] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:29.182317 23333 leader_election.cc:343] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: a2dbb7997d0f4e6bac48cfb196f5ccb6
I20251023 20:15:29.185062 23215 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 63) took 3154 ms. Trace:
I20251023 20:15:29.185190 23215 rpcz_store.cc:276] 1023 20:15:26.028742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.171104 (+3142362us) service_pool.cc:225] Handling call
1023 20:15:29.172107 (+ 1003us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:29.172554 (+ 447us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:29.183305 (+ 10751us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:29.183404 (+ 99us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:29.183493 (+ 89us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:29.173105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:29.173204 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:29.173211 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:29.173214 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:29.173296 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:29.173310 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:29.173315 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:29.173387 (+ 72us) tablet.cc:699] Row locks acquired
1023 20:15:29.173390 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:29.173425 (+ 35us) write_op.cc:270] Start()
1023 20:15:29.173472 (+ 47us) write_op.cc:276] Timestamp: P: 1761250529173398 usec, L: 0
1023 20:15:29.173476 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:29.173658 (+ 182us) log.cc:844] Serialized 433 byte log entry
1023 20:15:29.181968 (+ 8310us) op_driver.cc:464] REPLICATION: finished
1023 20:15:29.182114 (+ 146us) write_op.cc:301] APPLY: starting
1023 20:15:29.182131 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:29.182176 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:29.182209 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:29.182320 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:29.182355 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:29.182362 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:29.182444 (+ 82us) log.cc:844] Serialized 29 byte log entry
1023 20:15:29.182532 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.182553 (+ 21us) write_op.cc:454] Released schema lock
1023 20:15:29.182560 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:29.182642 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.182646 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":518,"num_ops":2,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":680,"prepare.run_wall_time_us":685,"raft.queue_time_us":151,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":223,"replication_time_us":8460}]]}
I20251023 20:15:29.186662 24349 tablet_replica.cc:333] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.187008 24349 raft_consensus.cc:2243] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.187501 24349 raft_consensus.cc:2272] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.188992 24349 ts_tablet_manager.cc:1916] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.189137 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_d55345cd0dd44edaa55e9e412037691f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:29.189800 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d55345cd0dd44edaa55e9e412037691f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:29.190601 23333 consensus_peers.cc:597] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.197474 24658 raft_consensus.cc:493] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.197546 23215 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 64) took 2809 ms. Trace:
I20251023 20:15:29.197633 24349 ts_tablet_manager.cc:1929] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.197719 24349 log.cc:1199] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/5e73ae95c73945f081c298f49d47c904
I20251023 20:15:29.197638 24658 raft_consensus.cc:515] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.198030 23206 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ea9be2a866cf46cfbec8db23a6d3f572 consensus state has no leader
I20251023 20:15:29.197633 23215 rpcz_store.cc:276] 1023 20:15:26.388291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.185753 (+2797462us) service_pool.cc:225] Handling call
1023 20:15:29.189947 (+ 4194us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:29.190351 (+ 404us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:29.197235 (+ 6884us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:29.197380 (+ 145us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:29.197479 (+ 99us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:29.190956 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:29.191068 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:29.191076 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:29.191078 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:29.191170 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:29.191183 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:29.191189 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:29.191261 (+ 72us) tablet.cc:699] Row locks acquired
1023 20:15:29.191263 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:29.191290 (+ 27us) write_op.cc:270] Start()
1023 20:15:29.191319 (+ 29us) write_op.cc:276] Timestamp: P: 1761250529191265 usec, L: 0
1023 20:15:29.191323 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:29.191472 (+ 149us) log.cc:844] Serialized 433 byte log entry
1023 20:15:29.194751 (+ 3279us) op_driver.cc:464] REPLICATION: finished
1023 20:15:29.194896 (+ 145us) write_op.cc:301] APPLY: starting
1023 20:15:29.194909 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:29.194952 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:29.194982 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:29.195082 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:29.195113 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:29.195120 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:29.195201 (+ 81us) log.cc:844] Serialized 29 byte log entry
1023 20:15:29.195262 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.195281 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:29.195286 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:29.195349 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.195353 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":446,"apply.run_wall_time_us":446,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":619,"prepare.run_wall_time_us":619,"raft.queue_time_us":129,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":209,"replication_time_us":3402}]]}
I20251023 20:15:29.198242 24658 leader_election.cc:290] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.198361 23215 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_9c3cb66b734f447e82a09d9e653ea072" } new_table_name: "default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18" modify_external_catalogs: true
I20251023 20:15:29.198760 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a54470d088748b0be22ab9db5a25f87" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.199240 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a54470d088748b0be22ab9db5a25f87" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.199298 23605 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:29.199729 23604 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.199877 23604 leader_election.cc:304] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.200348 24658 raft_consensus.cc:2749] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.204329 24349 ts_tablet_manager.cc:1950] T 5e73ae95c73945f081c298f49d47c904 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.204984 24830 raft_consensus.cc:493] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.205132 24830 raft_consensus.cc:515] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.205703 24830 leader_election.cc:290] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.206436 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ae910ac41fd4e349f159ab27e44375a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.206643 23395 raft_consensus.cc:2393] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:29.207789 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 5e73ae95c73945f081c298f49d47c904 (table default.Table_8d8ada20a7bb43eb9a534aedca23cfd2 [id=6fb3e731496a466a918ab84247724a22]) successfully deleted
I20251023 20:15:29.208456 24349 tablet_replica.cc:333] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.208901 24349 raft_consensus.cc:2243] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.209559 24349 raft_consensus.cc:2272] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.211107 23333 consensus_peers.cc:597] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e73ae95c73945f081c298f49d47c904. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.212916 24349 ts_tablet_manager.cc:1916] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.214726 24822 raft_consensus.cc:493] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.214882 24822 raft_consensus.cc:515] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.215411 24822 leader_election.cc:290] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.216163 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca01bea2bb146ca8797a9592a840a81" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.216673 23333 leader_election.cc:343] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.218844 24830 raft_consensus.cc:493] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.218969 24830 raft_consensus.cc:515] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:29.219486 24830 leader_election.cc:290] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.220297 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0858358aabff47a69dc77f4669b225c5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.220460 23395 raft_consensus.cc:2468] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:29.220809 23471 leader_election.cc:304] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:29.221043 24830 raft_consensus.cc:2804] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.221146 24830 raft_consensus.cc:493] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.221200 24830 raft_consensus.cc:3060] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.223625 24830 raft_consensus.cc:515] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:29.224495 24830 leader_election.cc:290] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.225736 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0858358aabff47a69dc77f4669b225c5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:29.225896 23395 raft_consensus.cc:3060] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.228627 23395 raft_consensus.cc:2468] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:29.229743 23471 leader_election.cc:304] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:29.231171 23333 consensus_peers.cc:597] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a27f192792cf41288d69ca15ab538472. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.231761 24830 raft_consensus.cc:2804] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.231895 24830 raft_consensus.cc:697] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:29.232878 24830 consensus_queue.cc:237] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:29.258100 23790 tablet_bootstrap.cc:492] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.258266 23790 ts_tablet_manager.cc:1403] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.166s user 0.004s sys 0.007s
I20251023 20:15:29.259014 23790 raft_consensus.cc:359] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.259187 23790 raft_consensus.cc:385] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.259234 23790 raft_consensus.cc:740] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.259449 23790 consensus_queue.cc:260] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.263147 23790 ts_tablet_manager.cc:1434] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:29.263592 23790 tablet_bootstrap.cc:492] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.265332 23790 tablet_bootstrap.cc:654] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.266327 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.266520 23664 raft_consensus.cc:3060] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.267750 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca01bea2bb146ca8797a9592a840a81" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.267910 23663 raft_consensus.cc:2468] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:29.268326 23470 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:29.268446 23470 leader_election.cc:304] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:29.268637 23787 tablet_bootstrap.cc:492] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:29.268837 23470 leader_election.cc:343] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 0858358aabff47a69dc77f4669b225c5
I20251023 20:15:29.268980 23664 raft_consensus.cc:2468] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:29.269116 23470 leader_election.cc:343] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 0858358aabff47a69dc77f4669b225c5
I20251023 20:15:29.269680 23333 leader_election.cc:304] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.269872 24363 tablet_replica.cc:333] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.270054 24363 raft_consensus.cc:2243] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:29.270162 23333 leader_election.cc:343] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a2dbb7997d0f4e6bac48cfb196f5ccb6
I20251023 20:15:29.270190 24363 raft_consensus.cc:2272] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.270251 23333 leader_election.cc:304] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.270581 24822 raft_consensus.cc:2804] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.270641 24821 raft_consensus.cc:2749] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:29.270655 24822 raft_consensus.cc:493] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.270903 24822 raft_consensus.cc:3060] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.272189 24363 ts_tablet_manager.cc:1916] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.273597 24780 ts_tablet_manager.cc:1929] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.273696 24780 log.cc:1199] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/aa794b3bec4b4b3993c814ea7cd02ee2
I20251023 20:15:29.273988 24780 ts_tablet_manager.cc:1950] T aa794b3bec4b4b3993c814ea7cd02ee2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.274998 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet aa794b3bec4b4b3993c814ea7cd02ee2 (table default.Table_083a66aed4484a38961dd79edd4d1c2e [id=a431259b5e254da6941241ffcf8ce597]) successfully deleted
I20251023 20:15:29.274968 24822 raft_consensus.cc:515] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.275323 24780 tablet_replica.cc:333] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.275616 24822 leader_election.cc:290] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.276016 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca01bea2bb146ca8797a9592a840a81" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.276049 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ca01bea2bb146ca8797a9592a840a81" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.276172 23670 raft_consensus.cc:3060] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:29.276580 23333 leader_election.cc:343] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.275499 24780 raft_consensus.cc:2243] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.276939 24780 pending_rounds.cc:70] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.276980 24780 pending_rounds.cc:77] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082166556971008 op_type: NO_OP noop_request { }
I20251023 20:15:29.277052 24780 raft_consensus.cc:2889] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.277146 24780 raft_consensus.cc:2272] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.268211 24349 ts_tablet_manager.cc:1929] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.277592 24349 log.cc:1199] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/f908f086946949829cac15418a584730
I20251023 20:15:29.277874 24349 ts_tablet_manager.cc:1950] T f908f086946949829cac15418a584730 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.277877 24780 ts_tablet_manager.cc:1916] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.278473 23670 raft_consensus.cc:2468] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.278863 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet f908f086946949829cac15418a584730 (table default.Table_a1c9b5357e504ba2a116c58ff86ad13c [id=25a11be83f484fa6abecb21ce1145e1e]) successfully deleted
I20251023 20:15:29.279029 23333 leader_election.cc:304] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.279246 24822 raft_consensus.cc:2804] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.279433 24822 raft_consensus.cc:697] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.279645 24822 consensus_queue.cc:237] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.280282 23333 leader_election.cc:304] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:29.280689 23333 consensus_peers.cc:597] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15ced74d0dd74e5195ed98358957d0dd. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.280841 24821 raft_consensus.cc:2804] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.280933 24821 raft_consensus.cc:697] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.281154 24821 consensus_queue.cc:237] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.281456 24830 raft_consensus.cc:3060] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.283484 24363 ts_tablet_manager.cc:1929] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.283636 24363 log.cc:1199] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/fb01b3f4ff7f41e98063ebf76abeace6
I20251023 20:15:29.283777 24830 raft_consensus.cc:2749] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.283967 24363 ts_tablet_manager.cc:1950] T fb01b3f4ff7f41e98063ebf76abeace6 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.284878 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet fb01b3f4ff7f41e98063ebf76abeace6 (table default.Table_0843993aca2340378776a12d7b3a5d4c [id=04fe1d72c1d64f14a77cb4f66a6ad8e0]) successfully deleted
I20251023 20:15:29.285229 24363 tablet_replica.cc:333] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.285435 24363 raft_consensus.cc:2243] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.286165 24363 raft_consensus.cc:2272] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.286929 24363 ts_tablet_manager.cc:1916] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.288251 23786 tablet_bootstrap.cc:492] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.288977 23786 ts_tablet_manager.cc:1403] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.137s user 0.014s sys 0.012s
I20251023 20:15:29.289790 23786 raft_consensus.cc:359] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.290896 23787 tablet_bootstrap.cc:654] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.291323 24659 raft_consensus.cc:493] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.291441 24659 raft_consensus.cc:515] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.291945 24659 leader_election.cc:290] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.292963 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25079532547d4175802b84877790662d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.294330 23790 tablet_bootstrap.cc:492] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.294517 23790 ts_tablet_manager.cc:1403] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20251023 20:15:29.296232 23790 raft_consensus.cc:359] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.296325 23605 leader_election.cc:343] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.296679 23790 raft_consensus.cc:385] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.296800 23790 raft_consensus.cc:740] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.292528 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25079532547d4175802b84877790662d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.297465 23604 leader_election.cc:343] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.297565 23604 leader_election.cc:304] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.298024 24659 raft_consensus.cc:2749] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.298247 23786 raft_consensus.cc:385] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.298593 23790 consensus_queue.cc:260] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.298822 23786 raft_consensus.cc:740] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.296607 24349 tablet_replica.cc:333] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.299175 24349 raft_consensus.cc:2243] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.299242 24363 ts_tablet_manager.cc:1929] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.299345 24349 raft_consensus.cc:2272] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.301559 23790 ts_tablet_manager.cc:1434] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251023 20:15:29.299513 23786 consensus_queue.cc:260] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.302431 24363 log.cc:1199] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/7e08dad1d0f643efb3a821a1b9196290
I20251023 20:15:29.303494 24363 ts_tablet_manager.cc:1950] T 7e08dad1d0f643efb3a821a1b9196290 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.304669 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 7e08dad1d0f643efb3a821a1b9196290 (table default.Table_ee55921ad9e8422984b6afd71da0f4ee [id=c6c3d95bdfef4028a356f1365e9fa53d]) successfully deleted
I20251023 20:15:29.305330 23787 tablet_bootstrap.cc:492] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.305708 23787 ts_tablet_manager.cc:1403] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20251023 20:15:29.306468 23787 raft_consensus.cc:359] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.306739 23787 raft_consensus.cc:385] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.306835 23787 raft_consensus.cc:740] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.306900 24363 tablet_replica.cc:333] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.307274 24363 raft_consensus.cc:2243] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.307299 23786 ts_tablet_manager.cc:1434] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251023 20:15:29.303325 24822 raft_consensus.cc:493] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:29.305557 23333 consensus_peers.cc:597] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af899c01b4af46d39d27f984cc3d5940. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.307531 24822 raft_consensus.cc:515] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.307662 23790 tablet_bootstrap.cc:492] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.308091 24822 leader_election.cc:290] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.308537 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352812b56f8d4feb818a364d6adfd86d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.308683 23670 raft_consensus.cc:2468] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:29.308749 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352812b56f8d4feb818a364d6adfd86d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.308372 24363 raft_consensus.cc:2272] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.302465 24349 ts_tablet_manager.cc:1916] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.309335 23333 leader_election.cc:304] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:29.309422 24363 ts_tablet_manager.cc:1916] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.309567 24822 raft_consensus.cc:2804] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.309698 24822 raft_consensus.cc:493] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.309813 24822 raft_consensus.cc:3060] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.307088 23787 consensus_queue.cc:260] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.309796 23333 leader_election.cc:343] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.311553 23786 tablet_bootstrap.cc:492] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.313223 24822 raft_consensus.cc:515] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.316433 24659 raft_consensus.cc:493] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.316545 24659 raft_consensus.cc:515] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.316890 23787 ts_tablet_manager.cc:1434] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251023 20:15:29.317056 24659 leader_election.cc:290] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.317191 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352812b56f8d4feb818a364d6adfd86d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.317432 23790 tablet_bootstrap.cc:654] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:29.317605 23333 leader_election.cc:343] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.317986 24822 leader_election.cc:290] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.317960 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:29.318516 23605 leader_election.cc:343] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.319169 23787 tablet_bootstrap.cc:492] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.319667 23786 tablet_bootstrap.cc:654] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.320113 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "352812b56f8d4feb818a364d6adfd86d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.320269 23665 raft_consensus.cc:3060] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.320941 23787 tablet_bootstrap.cc:654] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.322801 23665 raft_consensus.cc:2468] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.323441 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.323621 24349 ts_tablet_manager.cc:1929] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.323720 24349 log.cc:1199] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/1ae910ac41fd4e349f159ab27e44375a
W20251023 20:15:29.323889 23604 leader_election.cc:343] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.323975 24349 ts_tablet_manager.cc:1950] T 1ae910ac41fd4e349f159ab27e44375a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.323994 23604 leader_election.cc:304] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.324312 24659 raft_consensus.cc:2749] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.324581 23333 leader_election.cc:304] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.324841 24670 raft_consensus.cc:2804] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.324951 24670 raft_consensus.cc:697] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.325731 24670 consensus_queue.cc:237] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.339062 24349 tablet_replica.cc:333] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.339671 24349 raft_consensus.cc:2243] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.340845 24349 raft_consensus.cc:2272] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.340958 24659 raft_consensus.cc:493] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.341116 24659 raft_consensus.cc:515] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.341717 24659 leader_election.cc:290] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.342408 24349 ts_tablet_manager.cc:1916] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.344151 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a86a25659ec44d7c825783036342ed45" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.344815 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a86a25659ec44d7c825783036342ed45" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.345327 23604 leader_election.cc:343] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:29.346850 23605 leader_election.cc:343] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.347048 23605 leader_election.cc:304] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.347463 24659 raft_consensus.cc:2749] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:29.347653 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f because the tablet was not found. No further retry: Not found: Tablet not found: ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:29.349694 24780 ts_tablet_manager.cc:1929] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.349812 24780 log.cc:1199] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/48436f3d8f1443db990a2d10b668b70a
I20251023 20:15:29.350143 24780 ts_tablet_manager.cc:1950] T 48436f3d8f1443db990a2d10b668b70a P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.352742 24349 ts_tablet_manager.cc:1929] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.353295 23787 tablet_bootstrap.cc:492] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.353444 23790 tablet_bootstrap.cc:492] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.353437 23787 ts_tablet_manager.cc:1403] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251023 20:15:29.353610 23790 ts_tablet_manager.cc:1403] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.001s
I20251023 20:15:29.354205 23787 raft_consensus.cc:359] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.354374 23787 raft_consensus.cc:385] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.354434 23787 raft_consensus.cc:740] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.354305 23790 raft_consensus.cc:359] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.354636 23790 raft_consensus.cc:385] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.354729 23790 raft_consensus.cc:740] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.354674 23787 consensus_queue.cc:260] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.355008 23790 consensus_queue.cc:260] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.355537 23790 ts_tablet_manager.cc:1434] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:29.355840 23790 tablet_bootstrap.cc:492] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.357336 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 1ae910ac41fd4e349f159ab27e44375a (table default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]) successfully deleted
W20251023 20:15:29.358196 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 86b9f4c5d9e0469c87071a33722ce3eb because the tablet was not found. No further retry: Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb
W20251023 20:15:29.358453 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf because the tablet was not found. No further retry: Not found: Tablet not found: 1e8eb7603c3e46f1b15cdff7a7bbedcf
W20251023 20:15:29.358995 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2fa6a838b1bd409ba2a4e2074576ee23 because the tablet was not found. No further retry: Not found: Tablet not found: 2fa6a838b1bd409ba2a4e2074576ee23
W20251023 20:15:29.359193 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 because the tablet was not found. No further retry: Not found: Tablet not found: 7301d6121c9d4c8dbc642e895aa4e9f6
W20251023 20:15:29.359402 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
W20251023 20:15:29.359570 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a27f192792cf41288d69ca15ab538472 because the tablet was not found. No further retry: Not found: Tablet not found: a27f192792cf41288d69ca15ab538472
W20251023 20:15:29.359730 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9fc7e18054c440098dc2470143ea7fae because the tablet was not found. No further retry: Not found: Tablet not found: 9fc7e18054c440098dc2470143ea7fae
W20251023 20:15:29.359880 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
I20251023 20:15:29.360761 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 48436f3d8f1443db990a2d10b668b70a (table default.Table_7c03bf4f1b034b139999af6a349dccf0 [id=581d16998437467693ce84962d6717c7]) successfully deleted
I20251023 20:15:29.360944 24780 tablet_replica.cc:333] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
W20251023 20:15:29.361044 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a8eaa2167f82475ca2f6a110aba288ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8eaa2167f82475ca2f6a110aba288ec already in progress: creating tablet
I20251023 20:15:29.362365 23787 ts_tablet_manager.cc:1434] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:29.364776 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:29.364964 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:29.369894 23787 tablet_bootstrap.cc:492] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:29.370894 23333 consensus_peers.cc:597] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e73ae95c73945f081c298f49d47c904. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.372430 23790 tablet_bootstrap.cc:654] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.362715 23529 raft_consensus.cc:3060] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.373229 23787 tablet_bootstrap.cc:654] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.374377 24780 raft_consensus.cc:2243] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.374853 24780 pending_rounds.cc:70] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.374897 24780 pending_rounds.cc:77] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082168120410112 op_type: NO_OP noop_request { }
I20251023 20:15:29.374974 24780 raft_consensus.cc:2889] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.375039 24780 raft_consensus.cc:2272] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.372596 23786 tablet_bootstrap.cc:492] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.377904 23786 ts_tablet_manager.cc:1403] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.008s
I20251023 20:15:29.378166 24780 ts_tablet_manager.cc:1916] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.378935 23786 raft_consensus.cc:359] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.379845 23786 raft_consensus.cc:385] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.380015 23786 raft_consensus.cc:740] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.380398 23786 consensus_queue.cc:260] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.382126 23786 ts_tablet_manager.cc:1434] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:29.383139 23786 tablet_bootstrap.cc:492] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.385731 24349 log.cc:1199] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a8ba7ab28dfa4ac38e0b58ba2115495c
I20251023 20:15:29.386596 24349 ts_tablet_manager.cc:1950] T a8ba7ab28dfa4ac38e0b58ba2115495c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.387418 24780 ts_tablet_manager.cc:1929] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.387530 24780 log.cc:1199] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/352812b56f8d4feb818a364d6adfd86d
I20251023 20:15:29.387837 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a8ba7ab28dfa4ac38e0b58ba2115495c (table default.Table_4e18a968658f470d870bc932102ebf00 [id=ad634b38921149658d6212b02c421514]) successfully deleted
I20251023 20:15:29.387892 24780 ts_tablet_manager.cc:1950] T 352812b56f8d4feb818a364d6adfd86d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.387144 23786 tablet_bootstrap.cc:654] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.387815 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:29.388486 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:29.388592 24821 raft_consensus.cc:493] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.388705 24821 raft_consensus.cc:515] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.389287 24821 leader_election.cc:290] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.389477 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 352812b56f8d4feb818a364d6adfd86d (table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]) successfully deleted
I20251023 20:15:29.389705 24780 tablet_replica.cc:333] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.389743 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46f700ea3db04a4bad539c943e850a44" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.389874 24780 raft_consensus.cc:2243] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.389876 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46f700ea3db04a4bad539c943e850a44" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.389906 23664 raft_consensus.cc:2468] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:29.390079 24780 pending_rounds.cc:70] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.390127 24780 pending_rounds.cc:77] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082167100604416 op_type: NO_OP noop_request { }
I20251023 20:15:29.390201 24780 raft_consensus.cc:2889] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.390278 24780 raft_consensus.cc:2272] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.390297 23333 leader_election.cc:343] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.390455 23333 leader_election.cc:304] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.390622 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:29.390928 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
W20251023 20:15:29.391585 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
I20251023 20:15:29.391757 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:29.392916 24670 raft_consensus.cc:2804] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.393044 24670 raft_consensus.cc:493] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.393183 24670 raft_consensus.cc:3060] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.395707 24670 raft_consensus.cc:515] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.392309 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
W20251023 20:15:29.396462 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
I20251023 20:15:29.396545 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
W20251023 20:15:29.396729 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7540354f8577482785e6341fa9022315 because the tablet was not found. No further retry: Not found: Tablet not found: 7540354f8577482785e6341fa9022315
W20251023 20:15:29.396934 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
W20251023 20:15:29.397151 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 46f700ea3db04a4bad539c943e850a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f700ea3db04a4bad539c943e850a44 already in progress: creating tablet
I20251023 20:15:29.397218 24780 ts_tablet_manager.cc:1916] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.396736 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:29.397310 24821 raft_consensus.cc:493] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:29.397415 24821 raft_consensus.cc:515] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.397713 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:29.397881 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:29.397930 24821 leader_election.cc:290] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:29.398797 23333 leader_election.cc:343] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 8289ce37fa074891beb584bf2fff9747
W20251023 20:15:29.398970 23333 leader_election.cc:343] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 8289ce37fa074891beb584bf2fff9747
I20251023 20:15:29.399053 23333 leader_election.cc:304] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.399219 24670 leader_election.cc:290] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.399457 24670 raft_consensus.cc:2749] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:29.399730 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46f700ea3db04a4bad539c943e850a44" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.399756 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46f700ea3db04a4bad539c943e850a44" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.399912 23664 raft_consensus.cc:3060] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:29.400427 23333 leader_election.cc:343] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.400781 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:29.400991 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:29.401223 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:29.402447 23664 raft_consensus.cc:2468] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.402778 23333 leader_election.cc:304] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.402992 24822 raft_consensus.cc:2804] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.403092 24822 raft_consensus.cc:697] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.403288 24363 ts_tablet_manager.cc:1929] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.403380 24363 log.cc:1199] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/48436f3d8f1443db990a2d10b668b70a
I20251023 20:15:29.403306 24822 consensus_queue.cc:237] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.403680 24363 ts_tablet_manager.cc:1950] T 48436f3d8f1443db990a2d10b668b70a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:29.403434 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
W20251023 20:15:29.404148 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 8289ce37fa074891beb584bf2fff9747 because the tablet was not found. No further retry: Not found: Tablet not found: 8289ce37fa074891beb584bf2fff9747
I20251023 20:15:29.404415 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:29.404654 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:29.405221 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 48436f3d8f1443db990a2d10b668b70a (table default.Table_7c03bf4f1b034b139999af6a349dccf0 [id=581d16998437467693ce84962d6717c7]) successfully deleted
I20251023 20:15:29.409220 24363 tablet_replica.cc:333] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.409392 24363 raft_consensus.cc:2243] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.409538 24363 raft_consensus.cc:2272] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.410043 24363 ts_tablet_manager.cc:1916] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:29.411553 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
W20251023 20:15:29.411816 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
I20251023 20:15:29.411860 24297 raft_consensus.cc:493] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.411962 24297 raft_consensus.cc:515] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.412046 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 8317a54a5f7b4afe8da9d336268efe43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8317a54a5f7b4afe8da9d336268efe43 already in progress: creating tablet
W20251023 20:15:29.412482 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e19859a65a144598e1612be89daa3a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e19859a65a144598e1612be89daa3a1 already in progress: creating tablet
I20251023 20:15:29.412482 24297 leader_election.cc:290] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:29.412695 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
I20251023 20:15:29.413434 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
W20251023 20:15:29.413753 23333 leader_election.cc:343] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 523d72012c364457b9a2a299f2f789a1
I20251023 20:15:29.414177 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:29.414400 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:29.414707 23210 catalog_manager.cc:5649] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:29.415441 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:29.416200 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:29.416379 24780 ts_tablet_manager.cc:1929] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.416483 24780 log.cc:1199] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/f103bece2fa240c19c23ac51046c13ed
I20251023 20:15:29.416714 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:29.416821 24780 ts_tablet_manager.cc:1950] T f103bece2fa240c19c23ac51046c13ed P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.417323 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:29.418372 23786 tablet_bootstrap.cc:492] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.418931 23786 ts_tablet_manager.cc:1403] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20251023 20:15:29.420967 24780 tablet_replica.cc:333] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.421173 24780 raft_consensus.cc:2243] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.421388 24780 pending_rounds.cc:70] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.421438 24780 pending_rounds.cc:77] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082167389925376 op_type: NO_OP noop_request { }
I20251023 20:15:29.421541 24780 raft_consensus.cc:2889] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.421624 24780 raft_consensus.cc:2272] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.422941 24363 ts_tablet_manager.cc:1929] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.423043 24363 log.cc:1199] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a86a25659ec44d7c825783036342ed45
W20251023 20:15:29.423326 23333 leader_election.cc:343] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 523d72012c364457b9a2a299f2f789a1
I20251023 20:15:29.423420 23333 leader_election.cc:304] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.423892 24297 raft_consensus.cc:2749] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:29.424419 23786 raft_consensus.cc:359] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.424585 23786 raft_consensus.cc:385] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.424642 23786 raft_consensus.cc:740] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.417826 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet f103bece2fa240c19c23ac51046c13ed (table default.Table_c7fa864846ef4acbba390d7f493a8b58 [id=071ca3495096421eb75fea3fb71890ad]) successfully deleted
W20251023 20:15:29.423655 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
W20251023 20:15:29.425618 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 585a9af140d9479fb83d13e68a62300c because the tablet was not found. No further retry: Not found: Tablet not found: 585a9af140d9479fb83d13e68a62300c
I20251023 20:15:29.425755 24780 ts_tablet_manager.cc:1916] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.425807 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
W20251023 20:15:29.425849 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 79e3d422a66f432789c0cf4a00e57af2 because the tablet was not found. No further retry: Not found: Tablet not found: 79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:29.426044 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:29.426254 24349 tablet_replica.cc:333] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.427129 24349 raft_consensus.cc:2243] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.427331 24349 raft_consensus.cc:2272] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.428257 24349 ts_tablet_manager.cc:1916] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.428901 24363 ts_tablet_manager.cc:1950] T a86a25659ec44d7c825783036342ed45 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.428536 23786 consensus_queue.cc:260] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.430500 24363 tablet_replica.cc:333] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.431037 24363 raft_consensus.cc:2243] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.431175 24363 raft_consensus.cc:2272] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.431635 24363 ts_tablet_manager.cc:1916] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.433616 23786 ts_tablet_manager.cc:1434] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251023 20:15:29.436025 24780 ts_tablet_manager.cc:1929] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.436132 24780 log.cc:1199] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a636a4cdeb404e9f806d8e166ea57938
I20251023 20:15:29.436470 24780 ts_tablet_manager.cc:1950] T a636a4cdeb404e9f806d8e166ea57938 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.439313 24659 consensus_queue.cc:1048] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:29.441857 24780 tablet_replica.cc:333] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.442071 24780 raft_consensus.cc:2243] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.442324 24780 pending_rounds.cc:70] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.442385 24780 pending_rounds.cc:77] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082166356512768 op_type: NO_OP noop_request { }
I20251023 20:15:29.442487 24780 raft_consensus.cc:2889] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.442577 24780 raft_consensus.cc:2272] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.443986 23786 tablet_bootstrap.cc:492] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.444481 24780 ts_tablet_manager.cc:1916] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.429914 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a86a25659ec44d7c825783036342ed45 (table default.Table_9899db7c169244f38fdb46b1fd263e1a [id=7c367be5410c4fae9bff6652c2f15532]) successfully deleted
I20251023 20:15:29.437547 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a636a4cdeb404e9f806d8e166ea57938 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]) successfully deleted
I20251023 20:15:29.446154 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:29.446815 23786 tablet_bootstrap.cc:654] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.447052 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:29.447455 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:29.448613 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:29.448868 23790 tablet_bootstrap.cc:492] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.449093 23790 ts_tablet_manager.cc:1403] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.000s
I20251023 20:15:29.449841 23790 raft_consensus.cc:359] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.450014 23790 raft_consensus.cc:385] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.450115 23790 raft_consensus.cc:740] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.450299 24780 ts_tablet_manager.cc:1929] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.450466 24780 log.cc:1199] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/600f0110f8bd4b05983d5922d871c266
I20251023 20:15:29.450429 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:29.450683 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:29.450942 24780 ts_tablet_manager.cc:1950] T 600f0110f8bd4b05983d5922d871c266 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.451610 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:29.452391 24363 ts_tablet_manager.cc:1929] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.452502 24363 log.cc:1199] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a8eaa2167f82475ca2f6a110aba288ec
I20251023 20:15:29.452795 24363 ts_tablet_manager.cc:1950] T a8eaa2167f82475ca2f6a110aba288ec P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.451953 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 600f0110f8bd4b05983d5922d871c266 (table default.Table_f7824cd559744860a83775305e47389c [id=4b9ecb1c28154b86bc2b6da54948416b]) successfully deleted
I20251023 20:15:29.453619 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:29.453709 23790 consensus_queue.cc:260] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.454111 24363 tablet_replica.cc:333] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.454207 23790 ts_tablet_manager.cc:1434] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251023 20:15:29.454308 24363 raft_consensus.cc:2243] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.453881 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a8eaa2167f82475ca2f6a110aba288ec (table default.Table_f4de4bcad2f34502a1b63889f5b5e58a [id=05efdf3ecd3d49d4bc4da65e6f8f5b9b]) successfully deleted
I20251023 20:15:29.454433 24363 raft_consensus.cc:2272] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.454910 24363 ts_tablet_manager.cc:1916] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.455152 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:29.455785 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:29.456353 24349 ts_tablet_manager.cc:1929] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.456707 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:29.456905 24349 log.cc:1199] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d951134f6773410386bc57f7f90310c3
I20251023 20:15:29.457247 23790 tablet_bootstrap.cc:492] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.457609 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:29.457928 24349 ts_tablet_manager.cc:1950] T d951134f6773410386bc57f7f90310c3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.459031 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d951134f6773410386bc57f7f90310c3 (table default.Table_deb8b605c159407ab25c527a038c8d44 [id=7313119e56d94c05b9d8a31709af6ff6]) successfully deleted
I20251023 20:15:29.459985 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
W20251023 20:15:29.461237 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 994a555b24e4432a84601c54d422aa5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994a555b24e4432a84601c54d422aa5d already in progress: creating tablet
I20251023 20:15:29.461565 24363 ts_tablet_manager.cc:1929] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.461678 24363 log.cc:1199] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/352812b56f8d4feb818a364d6adfd86d
I20251023 20:15:29.461962 24363 ts_tablet_manager.cc:1950] T 352812b56f8d4feb818a364d6adfd86d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:29.462498 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
I20251023 20:15:29.462917 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 352812b56f8d4feb818a364d6adfd86d (table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]) successfully deleted
I20251023 20:15:29.463244 24780 tablet_replica.cc:333] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.463478 24780 raft_consensus.cc:2243] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.463671 24780 raft_consensus.cc:2272] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.463793 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6e1b4748c0244b258960b1025da47562 because the tablet was not found. No further retry: Not found: Tablet not found: 6e1b4748c0244b258960b1025da47562
I20251023 20:15:29.462967 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:29.464242 24780 ts_tablet_manager.cc:1916] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:29.464736 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
I20251023 20:15:29.465806 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:29.466447 24363 tablet_replica.cc:333] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.466662 24363 raft_consensus.cc:2243] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.466820 24363 raft_consensus.cc:2272] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.467317 24363 ts_tablet_manager.cc:1916] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:29.466315 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
W20251023 20:15:29.468318 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9d1ddde7e3134e23b3a9356523a93753 because the tablet was not found. No further retry: Not found: Tablet not found: 9d1ddde7e3134e23b3a9356523a93753
W20251023 20:15:29.468504 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
W20251023 20:15:29.468685 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ae388da9e4f94ba5af82a67088ab7097 because the tablet was not found. No further retry: Not found: Tablet not found: ae388da9e4f94ba5af82a67088ab7097
W20251023 20:15:29.468885 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 0438593fbbdf47cebfb64121acad104a because the tablet was not found. No further retry: Not found: Tablet not found: 0438593fbbdf47cebfb64121acad104a
I20251023 20:15:29.469841 23210 catalog_manager.cc:5649] T 37c8403b53f04a8e97ffe058e59b425c P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:29.472442 24830 raft_consensus.cc:493] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.472450 24659 raft_consensus.cc:493] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.472564 24830 raft_consensus.cc:515] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.472573 24659 raft_consensus.cc:515] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.473024 24363 ts_tablet_manager.cc:1929] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.473114 24830 leader_election.cc:290] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.473137 24659 leader_election.cc:290] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.473162 24363 log.cc:1199] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/f103bece2fa240c19c23ac51046c13ed
I20251023 20:15:29.473423 24363 ts_tablet_manager.cc:1950] T f103bece2fa240c19c23ac51046c13ed P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.473984 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ae6a6e58274b8baf952eaaefde7127" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.474113 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ae6a6e58274b8baf952eaaefde7127" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:29.474491 23471 leader_election.cc:343] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 48436f3d8f1443db990a2d10b668b70a
W20251023 20:15:29.474622 23605 leader_election.cc:343] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:29.474813 23470 leader_election.cc:343] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 48436f3d8f1443db990a2d10b668b70a
I20251023 20:15:29.474905 23470 leader_election.cc:304] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
W20251023 20:15:29.475468 23604 leader_election.cc:343] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.475560 23604 leader_election.cc:304] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.475935 24659 raft_consensus.cc:2749] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.476164 24830 raft_consensus.cc:2749] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:29.470599 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5346a221c78342359eb6e418bd12cd24 because the tablet was not found. No further retry: Not found: Tablet not found: 5346a221c78342359eb6e418bd12cd24
W20251023 20:15:29.477334 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet edec01a63a3046d9ae5a372fc9068b52 because the tablet was not found. No further retry: Not found: Tablet not found: edec01a63a3046d9ae5a372fc9068b52
W20251023 20:15:29.477528 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c95db7b5b9fd46acae8ffa7119915907 because the tablet was not found. No further retry: Not found: Tablet not found: c95db7b5b9fd46acae8ffa7119915907
W20251023 20:15:29.477721 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 4beebee06e5341a29c65d0c3cf985ef4 because the tablet was not found. No further retry: Not found: Tablet not found: 4beebee06e5341a29c65d0c3cf985ef4
W20251023 20:15:29.477918 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 4ea63540149c4c02abb8ae51209060a3 because the tablet was not found. No further retry: Not found: Tablet not found: 4ea63540149c4c02abb8ae51209060a3
W20251023 20:15:29.478129 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a4f012469fa4e8e827e485287ad76dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a4f012469fa4e8e827e485287ad76dd already in progress: creating tablet
W20251023 20:15:29.478305 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 because the tablet was not found. No further retry: Not found: Tablet not found: a2dbb7997d0f4e6bac48cfb196f5ccb6
W20251023 20:15:29.478515 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 377de4ea0bd6489284fac6b0fab52c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377de4ea0bd6489284fac6b0fab52c8c already in progress: creating tablet
I20251023 20:15:29.479396 23210 catalog_manager.cc:5649] T d6b0a4db18df490a966e8c475ddd14f4 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:29.483342 23786 tablet_bootstrap.cc:492] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.483508 23786 ts_tablet_manager.cc:1403] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.005s
I20251023 20:15:29.477957 24780 ts_tablet_manager.cc:1929] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.484400 24780 log.cc:1199] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/8da82e6d24b14a129b91bb5298139d07
I20251023 20:15:29.484408 23786 raft_consensus.cc:359] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.484663 23786 raft_consensus.cc:385] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.484753 23786 raft_consensus.cc:740] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.484773 24780 ts_tablet_manager.cc:1950] T 8da82e6d24b14a129b91bb5298139d07 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.485030 23786 consensus_queue.cc:260] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.487457 24780 tablet_replica.cc:333] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.487668 24780 raft_consensus.cc:2243] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 2 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.488181 23786 ts_tablet_manager.cc:1434] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251023 20:15:29.489390 24780 pending_rounds.cc:70] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.489487 24780 pending_rounds.cc:77] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7214082166001491968 op_type: NO_OP noop_request { }
I20251023 20:15:29.489589 24780 raft_consensus.cc:2889] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.489696 24780 raft_consensus.cc:2272] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.483132 23790 tablet_bootstrap.cc:654] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.492218 24780 ts_tablet_manager.cc:1916] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.492901 23786 tablet_bootstrap.cc:492] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:29.480358 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 419e41d7a37f4b888cb242b089c28790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 419e41d7a37f4b888cb242b089c28790 already in progress: creating tablet
I20251023 20:15:29.495254 23787 tablet_bootstrap.cc:492] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.495383 23787 ts_tablet_manager.cc:1403] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.127s user 0.005s sys 0.000s
I20251023 20:15:29.495362 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet f103bece2fa240c19c23ac51046c13ed (table default.Table_c7fa864846ef4acbba390d7f493a8b58 [id=071ca3495096421eb75fea3fb71890ad]) successfully deleted
W20251023 20:15:29.495627 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d82c92e863b2470699cc822b8cb8c14d because the tablet was not found. No further retry: Not found: Tablet not found: d82c92e863b2470699cc822b8cb8c14d
W20251023 20:15:29.495846 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 52bc660b9da54c5d984381a267e1efa2 because the tablet was not found. No further retry: Not found: Tablet not found: 52bc660b9da54c5d984381a267e1efa2
W20251023 20:15:29.496017 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 33b5efb30b5c491c8eafe3ac25688843 because the tablet was not found. No further retry: Not found: Tablet not found: 33b5efb30b5c491c8eafe3ac25688843
W20251023 20:15:29.496201 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
I20251023 20:15:29.496142 23787 raft_consensus.cc:359] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.496299 23787 raft_consensus.cc:385] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.496347 23787 raft_consensus.cc:740] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:29.496371 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 83da09ee0d10435ba89a19dc378d34eb because the tablet was not found. No further retry: Not found: Tablet not found: 83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:29.486733 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 8da82e6d24b14a129b91bb5298139d07 (table default.Table_58731fce9c224ff0b59d4b241110e518 [id=9ec90b724d89464a90821dd52282f77c]) successfully deleted
W20251023 20:15:29.496657 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
I20251023 20:15:29.496598 23787 consensus_queue.cc:260] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.497743 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:29.499738 24363 tablet_replica.cc:333] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.499923 24363 raft_consensus.cc:2243] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.500083 24363 raft_consensus.cc:2272] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.500102 24670 raft_consensus.cc:493] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.500265 24670 raft_consensus.cc:515] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.500547 24363 ts_tablet_manager.cc:1916] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:29.500828 23333 consensus_peers.cc:597] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.501222 24670 leader_election.cc:290] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.501603 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac55e14745534a02a2f6a422cab6b149" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.502079 23787 ts_tablet_manager.cc:1434] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20251023 20:15:29.502135 23333 leader_election.cc:343] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:29.502328 23333 leader_election.cc:343] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ac55e14745534a02a2f6a422cab6b149
I20251023 20:15:29.502452 23333 leader_election.cc:304] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.502712 23787 tablet_bootstrap.cc:492] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.502880 24670 raft_consensus.cc:2749] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.503999 23786 tablet_bootstrap.cc:654] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.504163 23790 tablet_bootstrap.cc:492] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:29.497785 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
I20251023 20:15:29.504321 23790 ts_tablet_manager.cc:1403] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
W20251023 20:15:29.504463 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
W20251023 20:15:29.504721 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cff337a6e728489b99fa8c5d4dfece2b because the tablet was not found. No further retry: Not found: Tablet not found: cff337a6e728489b99fa8c5d4dfece2b
W20251023 20:15:29.504933 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 because the tablet was not found. No further retry: Not found: Tablet not found: 3fbad98d3ab24cc5b7726a92fba04cf8
I20251023 20:15:29.505106 23787 tablet_bootstrap.cc:654] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:29.505177 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
I20251023 20:15:29.505069 23790 raft_consensus.cc:359] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.505288 23790 raft_consensus.cc:385] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.505365 23790 raft_consensus.cc:740] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:29.505396 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
I20251023 20:15:29.505601 23790 consensus_queue.cc:260] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.506002 23210 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:29.506596 23790 ts_tablet_manager.cc:1434] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:29.506922 23790 tablet_bootstrap.cc:492] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.508466 24349 tablet_replica.cc:333] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.508553 23790 tablet_bootstrap.cc:654] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.509286 24349 raft_consensus.cc:2243] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.509236 24825 consensus_queue.cc:1048] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:29.507010 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet dafb9db710f94c328186e3179bf0b319 because the tablet was not found. No further retry: Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319
W20251023 20:15:29.509554 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5964da70989b4e6cb77fe5d0a52ce9cf because the tablet was not found. No further retry: Not found: Tablet not found: 5964da70989b4e6cb77fe5d0a52ce9cf
W20251023 20:15:29.509764 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3894dd4feeea4c16861a6c9520059474 because the tablet was not found. No further retry: Not found: Tablet not found: 3894dd4feeea4c16861a6c9520059474
W20251023 20:15:29.509960 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e0f8d8184a68453abd2792ccc7740aa1 because the tablet was not found. No further retry: Not found: Tablet not found: e0f8d8184a68453abd2792ccc7740aa1
W20251023 20:15:29.510151 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7645c36c983f438f8b1223256fd3ad2c because the tablet was not found. No further retry: Not found: Tablet not found: 7645c36c983f438f8b1223256fd3ad2c
I20251023 20:15:29.511844 24349 pending_rounds.cc:70] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Trying to abort 1 pending ops.
W20251023 20:15:29.511864 23604 consensus_peers.cc:597] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.512357 24349 pending_rounds.cc:77] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082167746813952 op_type: NO_OP noop_request { }
I20251023 20:15:29.513134 24349 raft_consensus.cc:2889] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.513281 24349 raft_consensus.cc:2272] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.514724 24349 ts_tablet_manager.cc:1916] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.516968 24363 ts_tablet_manager.cc:1929] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.518507 24780 ts_tablet_manager.cc:1929] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251023 20:15:29.518636 24780 log.cc:1199] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:29.519037 24780 ts_tablet_manager.cc:1950] T cec51c54ea534b1da4b1d170e79d9f1c P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.517062 24363 log.cc:1199] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/600f0110f8bd4b05983d5922d871c266
I20251023 20:15:29.520502 24363 ts_tablet_manager.cc:1950] T 600f0110f8bd4b05983d5922d871c266 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.521500 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 600f0110f8bd4b05983d5922d871c266 (table default.Table_f7824cd559744860a83775305e47389c [id=4b9ecb1c28154b86bc2b6da54948416b]) successfully deleted
I20251023 20:15:29.521888 24780 tablet_replica.cc:333] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.522123 24780 raft_consensus.cc:2243] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.522490 24780 raft_consensus.cc:2272] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.523083 24780 ts_tablet_manager.cc:1916] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.525270 24363 tablet_replica.cc:333] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.525470 24363 raft_consensus.cc:2243] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.525482 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet cec51c54ea534b1da4b1d170e79d9f1c (table default.Table_21f0d4fca62a4309816edd74c6b137bf [id=be3ef152e4fd4281b5caf942c8d404ab]) successfully deleted
I20251023 20:15:29.525619 24363 raft_consensus.cc:2272] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.526136 24363 ts_tablet_manager.cc:1916] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.526190 24825 raft_consensus.cc:493] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea6853efbbcc484eb7529c7e9b37a47a)
I20251023 20:15:29.526367 24825 raft_consensus.cc:515] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.527098 24825 leader_election.cc:290] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
W20251023 20:15:29.527968 23333 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 83da09ee0d10435ba89a19dc378d34eb
W20251023 20:15:29.528275 23333 leader_election.cc:343] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:29.528393 23333 leader_election.cc:304] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.528573 23790 tablet_bootstrap.cc:492] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.528710 23790 ts_tablet_manager.cc:1403] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251023 20:15:29.528951 24821 raft_consensus.cc:2749] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:29.529485 23790 raft_consensus.cc:359] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.529661 23790 raft_consensus.cc:385] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.529747 23790 raft_consensus.cc:740] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.529940 23790 consensus_queue.cc:260] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.530687 23790 ts_tablet_manager.cc:1434] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:29.530973 23790 tablet_bootstrap.cc:492] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.533059 23790 tablet_bootstrap.cc:654] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.533607 24349 ts_tablet_manager.cc:1929] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.535336 24349 log.cc:1199] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/0858358aabff47a69dc77f4669b225c5
I20251023 20:15:29.535862 24349 ts_tablet_manager.cc:1950] T 0858358aabff47a69dc77f4669b225c5 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.536427 24363 ts_tablet_manager.cc:1929] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.536547 24363 log.cc:1199] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a636a4cdeb404e9f806d8e166ea57938
I20251023 20:15:29.536819 24363 ts_tablet_manager.cc:1950] T a636a4cdeb404e9f806d8e166ea57938 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.537562 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 167) took 1005 ms. Trace:
I20251023 20:15:29.537653 24363 rpcz_store.cc:276] 1023 20:15:28.532549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.532738 (+ 189us) service_pool.cc:225] Handling call
1023 20:15:29.525213 (+992475us) ts_tablet_manager.cc:1151] Deleting tablet a636a4cdeb404e9f806d8e166ea57938
1023 20:15:29.525226 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.533413 (+ 8187us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.536334 (+ 2921us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.537531 (+ 1197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":992349}
I20251023 20:15:29.538129 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a636a4cdeb404e9f806d8e166ea57938 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]) successfully deleted
I20251023 20:15:29.538479 24780 ts_tablet_manager.cc:1929] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.538614 24780 log.cc:1199] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/ef1229b9e0384a82811f244d2bf4c8ce
I20251023 20:15:29.538931 24780 ts_tablet_manager.cc:1950] T ef1229b9e0384a82811f244d2bf4c8ce P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.539778 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 0858358aabff47a69dc77f4669b225c5 (table default.Table_c9ccf1142a9b449f91e99360376a7b22 [id=c36b140eefe1435d93b7ba48ca07e4a1]) successfully deleted
I20251023 20:15:29.539887 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet ef1229b9e0384a82811f244d2bf4c8ce (table default.Table_d668e9c8e1394c4486b749d97aa78127 [id=70e4fcb37c9b49a386bbba20ab40ca06]) successfully deleted
W20251023 20:15:29.540195 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 6a54470d088748b0be22ab9db5a25f87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a54470d088748b0be22ab9db5a25f87 already in progress: creating tablet
I20251023 20:15:29.540230 24780 tablet_replica.cc:333] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.540417 24780 raft_consensus.cc:2243] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.540596 24780 raft_consensus.cc:2272] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.540930 24363 tablet_replica.cc:333] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.541121 24363 raft_consensus.cc:2243] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:29.541219 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 66a2f3adb5d347a0adc198cd1ef93304 because the tablet was not found. No further retry: Not found: Tablet not found: 66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:29.541263 24363 raft_consensus.cc:2272] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.541692 23790 tablet_bootstrap.cc:492] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.541745 24363 ts_tablet_manager.cc:1916] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.541863 23790 ts_tablet_manager.cc:1403] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20251023 20:15:29.541939 24830 raft_consensus.cc:493] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.542075 24830 raft_consensus.cc:515] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.542496 24780 ts_tablet_manager.cc:1916] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.542706 24830 leader_election.cc:290] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.542698 23790 raft_consensus.cc:359] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.542896 23790 raft_consensus.cc:385] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.542982 23790 raft_consensus.cc:740] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:29.543059 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e418a62d00824cff89b37a7bace5c469 because the tablet was not found. No further retry: Not found: Tablet not found: e418a62d00824cff89b37a7bace5c469
I20251023 20:15:29.543211 23790 consensus_queue.cc:260] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.543277 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.543337 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f422575c7c4251879130501ab721fd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.543454 23665 raft_consensus.cc:2393] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:29.543704 23790 ts_tablet_manager.cc:1434] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251023 20:15:29.544059 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 58ae6a6e58274b8baf952eaaefde7127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58ae6a6e58274b8baf952eaaefde7127 already in progress: creating tablet
I20251023 20:15:29.544096 23471 leader_election.cc:304] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:29.544456 24830 raft_consensus.cc:3060] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.545682 23790 tablet_bootstrap.cc:492] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.546962 24830 raft_consensus.cc:2749] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:29.547251 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
I20251023 20:15:29.547641 24349 tablet_replica.cc:333] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.548010 24349 raft_consensus.cc:2243] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.548388 24349 raft_consensus.cc:2272] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.549242 24349 ts_tablet_manager.cc:1916] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.557179 24363 ts_tablet_manager.cc:1929] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.557322 24363 log.cc:1199] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:29.557611 24363 ts_tablet_manager.cc:1950] T 6a54470d088748b0be22ab9db5a25f87 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.558473 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 161) took 1026 ms. Trace:
I20251023 20:15:29.558583 24363 rpcz_store.cc:276] 1023 20:15:28.532008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.532819 (+ 811us) service_pool.cc:225] Handling call
1023 20:15:29.540879 (+1008060us) ts_tablet_manager.cc:1151] Deleting tablet 6a54470d088748b0be22ab9db5a25f87
1023 20:15:29.540889 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.554123 (+ 13234us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.557046 (+ 2923us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.558449 (+ 1403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007921}
I20251023 20:15:29.559134 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 6a54470d088748b0be22ab9db5a25f87 (table default.Table_bd447f978d1d4b8bad839b915907db66 [id=a48d0e1e007c40d59766c7d0ef778956]) successfully deleted
I20251023 20:15:29.559554 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 165) took 1027 ms. Trace:
I20251023 20:15:29.559643 24363 rpcz_store.cc:276] 1023 20:15:28.532408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.532878 (+ 470us) service_pool.cc:225] Handling call
1023 20:15:29.559425 (+1026547us) ts_tablet_manager.cc:1151] Deleting tablet cec51c54ea534b1da4b1d170e79d9f1c
1023 20:15:29.559438 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.559525 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026423}
W20251023 20:15:29.560101 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cec51c54ea534b1da4b1d170e79d9f1c because the tablet was not found. No further retry: Not found: Tablet not found: cec51c54ea534b1da4b1d170e79d9f1c
I20251023 20:15:29.560446 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 168) took 1027 ms. Trace:
I20251023 20:15:29.560528 24363 rpcz_store.cc:276] 1023 20:15:28.533346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.533392 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:29.560356 (+1026964us) ts_tablet_manager.cc:1151] Deleting tablet 523d72012c364457b9a2a299f2f789a1
1023 20:15:29.560367 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.560429 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026814}
W20251023 20:15:29.560947 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 523d72012c364457b9a2a299f2f789a1 because the tablet was not found. No further retry: Not found: Tablet not found: 523d72012c364457b9a2a299f2f789a1
I20251023 20:15:29.561245 24363 tablet_replica.cc:333] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.561447 24363 raft_consensus.cc:2243] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.561606 24363 raft_consensus.cc:2272] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.562139 24363 ts_tablet_manager.cc:1916] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.564447 24349 ts_tablet_manager.cc:1929] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.567744 24349 log.cc:1199] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/15ced74d0dd74e5195ed98358957d0dd
I20251023 20:15:29.567931 24363 ts_tablet_manager.cc:1929] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.568038 24363 log.cc:1199] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d9f422575c7c4251879130501ab721fd
I20251023 20:15:29.568248 23790 tablet_bootstrap.cc:654] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.568607 24349 ts_tablet_manager.cc:1950] T 15ced74d0dd74e5195ed98358957d0dd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.569805 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 15ced74d0dd74e5195ed98358957d0dd (table default.Table_e62c8689605f4db7b6e5016d5dcda175 [id=91e8d69e8f264c6dafd1173e05f7be06]) successfully deleted
I20251023 20:15:29.570164 24780 ts_tablet_manager.cc:1929] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.570329 24780 log.cc:1199] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:29.570734 24780 ts_tablet_manager.cc:1950] T d82c92e863b2470699cc822b8cb8c14d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.570981 24349 tablet_replica.cc:333] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.571369 24349 raft_consensus.cc:2243] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.571573 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 84) took 1007 ms. Trace:
I20251023 20:15:29.571800 24349 raft_consensus.cc:2272] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.571714 24780 rpcz_store.cc:276] 1023 20:15:28.563655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.563691 (+ 36us) service_pool.cc:225] Handling call
1023 20:15:29.540152 (+976461us) ts_tablet_manager.cc:1151] Deleting tablet d82c92e863b2470699cc822b8cb8c14d
1023 20:15:29.540160 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.550110 (+ 9950us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.570006 (+ 19896us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.571508 (+ 1502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":976335}
I20251023 20:15:29.572314 24780 tablet_replica.cc:333] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.572556 24780 raft_consensus.cc:2243] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.572817 24780 pending_rounds.cc:70] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.572907 24780 pending_rounds.cc:77] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082166149025792 op_type: NO_OP noop_request { }
I20251023 20:15:29.573421 24363 ts_tablet_manager.cc:1950] T d9f422575c7c4251879130501ab721fd P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.573390 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d82c92e863b2470699cc822b8cb8c14d (table default.Table_8ac87d9615a74c5c858be1d6c31772a6 [id=1740292e42a544e5bd5d834a281bddbc]) successfully deleted
I20251023 20:15:29.573652 24822 raft_consensus.cc:493] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:29.573630 24780 raft_consensus.cc:2889] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.573853 24822 raft_consensus.cc:515] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.574343 24822 leader_election.cc:290] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.574438 24822 raft_consensus.cc:493] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.574518 24822 raft_consensus.cc:515] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.574986 24822 leader_election.cc:290] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.576109 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "994a555b24e4432a84601c54d422aa5d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.576279 23670 raft_consensus.cc:2468] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:29.576700 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "994a555b24e4432a84601c54d422aa5d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.577378 23333 leader_election.cc:343] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 7540354f8577482785e6341fa9022315
W20251023 20:15:29.577558 23333 leader_election.cc:343] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.577584 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 169) took 1043 ms. Trace:
W20251023 20:15:29.577725 23333 leader_election.cc:343] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 7540354f8577482785e6341fa9022315
I20251023 20:15:29.577656 24363 rpcz_store.cc:276] 1023 20:15:28.533845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.533889 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:29.561191 (+1027302us) ts_tablet_manager.cc:1151] Deleting tablet d9f422575c7c4251879130501ab721fd
1023 20:15:29.561202 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.565051 (+ 3849us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.567854 (+ 2803us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.577565 (+ 9711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027168}
I20251023 20:15:29.577822 23333 leader_election.cc:304] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.577947 24363 tablet_replica.cc:333] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.578089 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d9f422575c7c4251879130501ab721fd (table default.Table_c01f9e9030974e2e8b43da4e68ef0145 [id=12899265e5544ac4837f4916f8bab077]) successfully deleted
I20251023 20:15:29.578116 24363 raft_consensus.cc:2243] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.578238 24363 raft_consensus.cc:2272] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.578199 23333 leader_election.cc:304] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.578578 24297 raft_consensus.cc:2749] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:29.578693 24363 ts_tablet_manager.cc:1916] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.578714 24670 raft_consensus.cc:2804] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.578783 24670 raft_consensus.cc:493] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.578835 24670 raft_consensus.cc:3060] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.581280 24670 raft_consensus.cc:515] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.581836 24670 leader_election.cc:290] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.582662 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "994a555b24e4432a84601c54d422aa5d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.582710 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "994a555b24e4432a84601c54d422aa5d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.582820 23670 raft_consensus.cc:3060] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:29.583249 23333 leader_election.cc:343] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.583652 24349 ts_tablet_manager.cc:1916] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.585191 23670 raft_consensus.cc:2468] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.585529 23333 leader_election.cc:304] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.586234 24822 raft_consensus.cc:2804] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.586347 24822 raft_consensus.cc:697] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.586587 24822 consensus_queue.cc:237] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:29.588805 23333 consensus_peers.cc:597] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.573875 24780 raft_consensus.cc:2272] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.590149 24780 ts_tablet_manager.cc:1916] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.592430 23787 tablet_bootstrap.cc:492] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.592623 23787 ts_tablet_manager.cc:1403] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.003s
I20251023 20:15:29.595960 24780 ts_tablet_manager.cc:1929] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.596096 24780 log.cc:1199] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d9f422575c7c4251879130501ab721fd
I20251023 20:15:29.596443 24780 ts_tablet_manager.cc:1950] T d9f422575c7c4251879130501ab721fd P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.597278 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 85) took 1033 ms. Trace:
I20251023 20:15:29.597368 24780 rpcz_store.cc:276] 1023 20:15:28.563948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.564508 (+ 560us) service_pool.cc:225] Handling call
1023 20:15:29.572233 (+1007725us) ts_tablet_manager.cc:1151] Deleting tablet d9f422575c7c4251879130501ab721fd
1023 20:15:29.572244 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.593174 (+ 20930us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.595862 (+ 2688us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.597255 (+ 1393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007568}
I20251023 20:15:29.597711 24780 tablet_replica.cc:333] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.597862 24780 raft_consensus.cc:2243] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.597834 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d9f422575c7c4251879130501ab721fd (table default.Table_c01f9e9030974e2e8b43da4e68ef0145 [id=12899265e5544ac4837f4916f8bab077]) successfully deleted
I20251023 20:15:29.598047 24780 pending_rounds.cc:70] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.598100 24780 pending_rounds.cc:77] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082165602246656 op_type: NO_OP noop_request { }
I20251023 20:15:29.598299 24780 raft_consensus.cc:2889] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.598390 24780 raft_consensus.cc:2272] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.598894 24670 raft_consensus.cc:493] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:29.599045 24670 raft_consensus.cc:515] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.599261 23787 raft_consensus.cc:359] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.599460 23787 raft_consensus.cc:385] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.599566 23787 raft_consensus.cc:740] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.599675 24670 leader_election.cc:290] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.599885 23787 consensus_queue.cc:260] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.600430 23787 ts_tablet_manager.cc:1434] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251023 20:15:29.600703 23333 leader_election.cc:343] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 0438593fbbdf47cebfb64121acad104a
W20251023 20:15:29.600888 23333 leader_election.cc:343] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 0438593fbbdf47cebfb64121acad104a
I20251023 20:15:29.600962 23333 leader_election.cc:304] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.601593 24670 raft_consensus.cc:2749] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:29.599040 24780 ts_tablet_manager.cc:1916] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.601758 23787 tablet_bootstrap.cc:492] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.601920 24363 ts_tablet_manager.cc:1929] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.602099 24363 log.cc:1199] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/83dae34129794f49b43038d3b1636ea0
I20251023 20:15:29.602412 24363 ts_tablet_manager.cc:1950] T 83dae34129794f49b43038d3b1636ea0 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.603160 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 170) took 1068 ms. Trace:
I20251023 20:15:29.603348 23787 tablet_bootstrap.cc:654] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.603295 24363 rpcz_store.cc:276] 1023 20:15:28.534277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.534321 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:29.577903 (+1043582us) ts_tablet_manager.cc:1151] Deleting tablet 83dae34129794f49b43038d3b1636ea0
1023 20:15:29.577913 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.598238 (+ 20325us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.601825 (+ 3587us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.603141 (+ 1316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043443}
I20251023 20:15:29.603897 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 83dae34129794f49b43038d3b1636ea0 (table default.Table_52af8fa4ff964f16b6fe2b43e94dddd4 [id=872a29a7d5234fc98b6dd83be48499cd]) successfully deleted
I20251023 20:15:29.604025 24670 raft_consensus.cc:493] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.604075 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 171) took 1069 ms. Trace:
I20251023 20:15:29.604149 24670 raft_consensus.cc:515] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.604640 24363 rpcz_store.cc:276] 1023 20:15:28.534800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.534840 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:29.603902 (+1069062us) ts_tablet_manager.cc:1151] Deleting tablet 1e5720c96730425dafb550094cf936cd
1023 20:15:29.603912 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.604003 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068925}
I20251023 20:15:29.605010 24670 leader_election.cc:290] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:29.605279 24363 tablet_replica.cc:333] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:29.605283 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
I20251023 20:15:29.605545 24363 raft_consensus.cc:2243] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:29.606022 23333 leader_election.cc:343] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: f908f086946949829cac15418a584730
I20251023 20:15:29.606173 24363 raft_consensus.cc:2272] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.606348 23333 leader_election.cc:343] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: f908f086946949829cac15418a584730
I20251023 20:15:29.606443 23333 leader_election.cc:304] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.606793 24670 raft_consensus.cc:2749] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.606844 24363 ts_tablet_manager.cc:1916] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.612962 24363 ts_tablet_manager.cc:1929] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.613050 23790 tablet_bootstrap.cc:492] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.613184 24363 log.cc:1199] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/a9c5b69df4c04365b1ac1457fe8300cc
I20251023 20:15:29.613212 23790 ts_tablet_manager.cc:1403] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.009s
I20251023 20:15:29.613561 24363 ts_tablet_manager.cc:1950] T a9c5b69df4c04365b1ac1457fe8300cc P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.613891 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 172) took 1078 ms. Trace:
I20251023 20:15:29.613986 23790 raft_consensus.cc:359] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.614138 23790 raft_consensus.cc:385] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.614189 23790 raft_consensus.cc:740] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.614009 24363 rpcz_store.cc:276] 1023 20:15:28.535230 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.535272 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:29.605232 (+1069960us) ts_tablet_manager.cc:1151] Deleting tablet a9c5b69df4c04365b1ac1457fe8300cc
1023 20:15:29.605241 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.609791 (+ 4550us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.612889 (+ 3098us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.613876 (+ 987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069833}
I20251023 20:15:29.614368 23790 consensus_queue.cc:260] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.614622 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet a9c5b69df4c04365b1ac1457fe8300cc (table default.Table_8c497d9a1dcc41aa8466741225b4de85 [id=5746055cb51b4ee4b54b672987471cb1]) successfully deleted
I20251023 20:15:29.614811 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 173) took 1078 ms. Trace:
I20251023 20:15:29.614916 24363 rpcz_store.cc:276] 1023 20:15:28.536123 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.536276 (+ 153us) service_pool.cc:225] Handling call
1023 20:15:29.614668 (+1078392us) ts_tablet_manager.cc:1151] Deleting tablet 7645c36c983f438f8b1223256fd3ad2c
1023 20:15:29.614677 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.614794 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078255}
I20251023 20:15:29.615109 23786 tablet_bootstrap.cc:492] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.615129 23790 ts_tablet_manager.cc:1434] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:29.615255 23786 ts_tablet_manager.cc:1403] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.016s
W20251023 20:15:29.615415 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7645c36c983f438f8b1223256fd3ad2c because the tablet was not found. No further retry: Not found: Tablet not found: 7645c36c983f438f8b1223256fd3ad2c
I20251023 20:15:29.615447 24931 raft_consensus.cc:493] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.615536 24780 ts_tablet_manager.cc:1929] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.615621 24780 log.cc:1199] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/c97e2e0f839a4ef389fe100b6a81301e
I20251023 20:15:29.615722 24931 raft_consensus.cc:515] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.616066 24780 ts_tablet_manager.cc:1950] T c97e2e0f839a4ef389fe100b6a81301e P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.616039 23786 raft_consensus.cc:359] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.616196 23786 raft_consensus.cc:385] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.616255 23786 raft_consensus.cc:740] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.616470 23786 consensus_queue.cc:260] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.616828 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 174) took 1080 ms. Trace:
I20251023 20:15:29.616917 23786 ts_tablet_manager.cc:1434] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:29.617129 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05933cb9433d48d285b2db82b289bdc8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.617199 23786 tablet_bootstrap.cc:492] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.617278 24363 rpcz_store.cc:276] 1023 20:15:28.536232 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.536495 (+ 263us) service_pool.cc:225] Handling call
1023 20:15:29.616740 (+1080245us) ts_tablet_manager.cc:1151] Deleting tablet 9d1ddde7e3134e23b3a9356523a93753
1023 20:15:29.616750 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.616814 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080116}
W20251023 20:15:29.617619 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9d1ddde7e3134e23b3a9356523a93753 because the tablet was not found. No further retry: Not found: Tablet not found: 9d1ddde7e3134e23b3a9356523a93753
I20251023 20:15:29.617211 24931 leader_election.cc:290] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.617935 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 176) took 1080 ms. Trace:
I20251023 20:15:29.617997 24363 rpcz_store.cc:276] 1023 20:15:28.537071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.537130 (+ 59us) service_pool.cc:225] Handling call
1023 20:15:29.617863 (+1080733us) ts_tablet_manager.cc:1151] Deleting tablet 33b5efb30b5c491c8eafe3ac25688843
1023 20:15:29.617871 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.617921 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080601}
W20251023 20:15:29.618368 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 33b5efb30b5c491c8eafe3ac25688843 because the tablet was not found. No further retry: Not found: Tablet not found: 33b5efb30b5c491c8eafe3ac25688843
I20251023 20:15:29.618510 24349 ts_tablet_manager.cc:1929] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.618592 24349 log.cc:1199] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/252b5480222f49439052c86a4115cc9b
I20251023 20:15:29.618628 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 175) took 1081 ms. Trace:
I20251023 20:15:29.618690 24363 rpcz_store.cc:276] 1023 20:15:28.536963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.537341 (+ 378us) service_pool.cc:225] Handling call
1023 20:15:29.618558 (+1081217us) ts_tablet_manager.cc:1151] Deleting tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0
1023 20:15:29.618566 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.618613 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081087}
I20251023 20:15:29.618888 24349 ts_tablet_manager.cc:1950] T 252b5480222f49439052c86a4115cc9b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:29.619038 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 because the tablet was not found. No further retry: Not found: Tablet not found: 3a4c8dcbdfcf4a15b8a4194ea04b93a0
I20251023 20:15:29.619319 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05933cb9433d48d285b2db82b289bdc8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.619467 24363 tablet_replica.cc:333] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.619633 24363 raft_consensus.cc:2243] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.619760 24363 raft_consensus.cc:2272] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.619788 23604 leader_election.cc:343] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.620239 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 252b5480222f49439052c86a4115cc9b (table default.Table_bdb5b8891a814c998413f5b7a0a605cc [id=0674b989e3f74438b47e1fc653f5db49]) successfully deleted
W20251023 20:15:29.620568 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
W20251023 20:15:29.620595 23605 leader_election.cc:343] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.620677 23605 leader_election.cc:304] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:29.620812 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 23225d31c2e647398aa58494c627507e because the tablet was not found. No further retry: Not found: Tablet not found: 23225d31c2e647398aa58494c627507e
W20251023 20:15:29.621013 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d0e61de7a93d450d9ed79eef82d7e5e3 because the tablet was not found. No further retry: Not found: Tablet not found: d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:29.621258 23786 tablet_bootstrap.cc:654] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:29.621261 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
W20251023 20:15:29.621469 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a21ec49194e24723b31fd9c86f1f85e4 because the tablet was not found. No further retry: Not found: Tablet not found: a21ec49194e24723b31fd9c86f1f85e4
W20251023 20:15:29.621707 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 40e0301fb5974ec5b6e8291230a736c2 because the tablet was not found. No further retry: Not found: Tablet not found: 40e0301fb5974ec5b6e8291230a736c2
W20251023 20:15:29.621940 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 25079532547d4175802b84877790662d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25079532547d4175802b84877790662d already in progress: creating tablet
W20251023 20:15:29.622156 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fe922c438a894a7aa04699b2d2897a28 because the tablet was not found. No further retry: Not found: Tablet not found: fe922c438a894a7aa04699b2d2897a28
W20251023 20:15:29.622375 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 because the tablet was not found. No further retry: Not found: Tablet not found: 283166bbd8004ce3bca9f2ecbdbcbf48
W20251023 20:15:29.622606 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
W20251023 20:15:29.622838 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2532c8567db14901b383f42e5f04b4c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2532c8567db14901b383f42e5f04b4c2
W20251023 20:15:29.623057 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ac55e14745534a02a2f6a422cab6b149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac55e14745534a02a2f6a422cab6b149 already in progress: creating tablet
W20251023 20:15:29.623260 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 because the tablet was not found. No further retry: Not found: Tablet not found: 3a4c8dcbdfcf4a15b8a4194ea04b93a0
W20251023 20:15:29.623479 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d62dbf044acd4459b26d2e7e50f6206e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d62dbf044acd4459b26d2e7e50f6206e already in progress: creating tablet
I20251023 20:15:29.623507 24349 tablet_replica.cc:333] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:29.623677 24349 raft_consensus.cc:2243] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:29.623764 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e5368813164949ff80e27bf59dcc931a because the tablet was not found. No further retry: Not found: Tablet not found: e5368813164949ff80e27bf59dcc931a
I20251023 20:15:29.623908 24349 raft_consensus.cc:2272] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.623975 24931 raft_consensus.cc:2749] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.624418 24363 ts_tablet_manager.cc:1916] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.625905 24349 ts_tablet_manager.cc:1916] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.631631 24363 ts_tablet_manager.cc:1929] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.631803 24363 log.cc:1199] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/46f700ea3db04a4bad539c943e850a44
I20251023 20:15:29.632121 24363 ts_tablet_manager.cc:1950] T 46f700ea3db04a4bad539c943e850a44 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.632198 24349 ts_tablet_manager.cc:1929] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:29.632334 24349 log.cc:1199] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d2ae19cc80194363b8a9c96fbafb4ebe
I20251023 20:15:29.632737 24349 ts_tablet_manager.cc:1950] T d2ae19cc80194363b8a9c96fbafb4ebe P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:29.632895 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 177) took 1049 ms. Trace:
I20251023 20:15:29.633018 24363 rpcz_store.cc:276] 1023 20:15:28.583879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.583943 (+ 64us) service_pool.cc:225] Handling call
1023 20:15:29.619427 (+1035484us) ts_tablet_manager.cc:1151] Deleting tablet 46f700ea3db04a4bad539c943e850a44
1023 20:15:29.619435 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.628864 (+ 9429us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.631555 (+ 2691us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.632878 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035319}
I20251023 20:15:29.633528 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 46f700ea3db04a4bad539c943e850a44 (table default.Table_aba84ed1ff684cc690f31c5615ec1700 [id=0b966d7b9d4d4ad1bdf1c84d63c6be5a]) successfully deleted
I20251023 20:15:29.633780 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]) successfully deleted
I20251023 20:15:29.634156 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 179) took 1042 ms. Trace:
I20251023 20:15:29.634153 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 86) took 1069 ms. Trace:
I20251023 20:15:29.634232 24780 rpcz_store.cc:276] 1023 20:15:28.564819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.565108 (+ 289us) service_pool.cc:225] Handling call
1023 20:15:29.597664 (+1032556us) ts_tablet_manager.cc:1151] Deleting tablet c97e2e0f839a4ef389fe100b6a81301e
1023 20:15:29.597675 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.609329 (+ 11654us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.615454 (+ 6125us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.634130 (+ 18676us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1032424}
I20251023 20:15:29.634282 24363 rpcz_store.cc:276] 1023 20:15:28.591698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.591807 (+ 109us) service_pool.cc:225] Handling call
1023 20:15:29.634053 (+1042246us) ts_tablet_manager.cc:1151] Deleting tablet 40c659c273c449939d0dd708d39b393f
1023 20:15:29.634062 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.634142 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041962}
I20251023 20:15:29.634608 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 87) took 1069 ms. Trace:
I20251023 20:15:29.634619 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet c97e2e0f839a4ef389fe100b6a81301e (table default.Table_5d14af3d31254be5a7ff6cb5d1fea162 [id=55a8a0300c454094a6ee9c2c7897495c]) successfully deleted
I20251023 20:15:29.634680 24780 rpcz_store.cc:276] 1023 20:15:28.565387 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.565671 (+ 284us) service_pool.cc:225] Handling call
1023 20:15:29.634483 (+1068812us) ts_tablet_manager.cc:1151] Deleting tablet cbacaf2fea24405caaf84d59b625fca3
1023 20:15:29.634492 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.634592 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068689}
W20251023 20:15:29.634830 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
I20251023 20:15:29.634935 24780 tablet_replica.cc:333] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
W20251023 20:15:29.635005 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet cbacaf2fea24405caaf84d59b625fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbacaf2fea24405caaf84d59b625fca3 already in progress: creating tablet
I20251023 20:15:29.635103 24780 raft_consensus.cc:2243] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.635332 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 181) took 1043 ms. Trace:
I20251023 20:15:29.635402 24780 raft_consensus.cc:2272] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.635447 24363 rpcz_store.cc:276] 1023 20:15:28.591885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.592541 (+ 656us) service_pool.cc:225] Handling call
1023 20:15:29.635221 (+1042680us) ts_tablet_manager.cc:1151] Deleting tablet 7540354f8577482785e6341fa9022315
1023 20:15:29.635229 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.635317 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042577}
W20251023 20:15:29.635939 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7540354f8577482785e6341fa9022315 because the tablet was not found. No further retry: Not found: Tablet not found: 7540354f8577482785e6341fa9022315
I20251023 20:15:29.636061 24780 ts_tablet_manager.cc:1916] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.636286 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 185) took 1044 ms. Trace:
I20251023 20:15:29.636394 24363 rpcz_store.cc:276] 1023 20:15:28.592157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.592205 (+ 48us) service_pool.cc:225] Handling call
1023 20:15:29.636205 (+1044000us) ts_tablet_manager.cc:1151] Deleting tablet ae388da9e4f94ba5af82a67088ab7097
1023 20:15:29.636213 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.636272 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":221,"tablet-delete.queue_time_us":1043758}
W20251023 20:15:29.636838 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ae388da9e4f94ba5af82a67088ab7097 because the tablet was not found. No further retry: Not found: Tablet not found: ae388da9e4f94ba5af82a67088ab7097
I20251023 20:15:29.637192 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 180) took 1045 ms. Trace:
I20251023 20:15:29.637296 24363 rpcz_store.cc:276] 1023 20:15:28.591808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.592396 (+ 588us) service_pool.cc:225] Handling call
1023 20:15:29.637113 (+1044717us) ts_tablet_manager.cc:1151] Deleting tablet 8289ce37fa074891beb584bf2fff9747
1023 20:15:29.637121 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.637178 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":209,"tablet-delete.queue_time_us":1044556}
W20251023 20:15:29.637712 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 8289ce37fa074891beb584bf2fff9747 because the tablet was not found. No further retry: Not found: Tablet not found: 8289ce37fa074891beb584bf2fff9747
I20251023 20:15:29.638060 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 183) took 1046 ms. Trace:
I20251023 20:15:29.638171 24363 rpcz_store.cc:276] 1023 20:15:28.592024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.592125 (+ 101us) service_pool.cc:225] Handling call
1023 20:15:29.637961 (+1045836us) ts_tablet_manager.cc:1151] Deleting tablet a841ffdc6cdd432dac7afe2e2b53f224
1023 20:15:29.637969 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.638045 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":185,"tablet-delete.queue_time_us":1045387}
W20251023 20:15:29.638607 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
I20251023 20:15:29.638983 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 178) took 1047 ms. Trace:
I20251023 20:15:29.640918 24363 rpcz_store.cc:276] 1023 20:15:28.591432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.592204 (+ 772us) service_pool.cc:225] Handling call
1023 20:15:29.638865 (+1046661us) ts_tablet_manager.cc:1151] Deleting tablet 01fa5f9c700b458ea147373d78551573
1023 20:15:29.638873 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.638966 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":395,"tablet-delete.queue_time_us":1046482}
W20251023 20:15:29.641618 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
I20251023 20:15:29.642756 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:29.643041 24780 ts_tablet_manager.cc:1929] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.643137 24780 log.cc:1199] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/1a64a7c2e1f242e1b54243489622bae5
I20251023 20:15:29.643440 24780 ts_tablet_manager.cc:1950] T 1a64a7c2e1f242e1b54243489622bae5 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.643553 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 184) took 1051 ms. Trace:
I20251023 20:15:29.643626 24363 rpcz_store.cc:276] 1023 20:15:28.592089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.592794 (+ 705us) service_pool.cc:225] Handling call
1023 20:15:29.643444 (+1050650us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:29.643454 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.643539 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048834}
I20251023 20:15:29.644121 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 88) took 1077 ms. Trace:
I20251023 20:15:29.644228 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 182) took 1052 ms. Trace:
I20251023 20:15:29.644200 24780 rpcz_store.cc:276] 1023 20:15:28.566192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.566470 (+ 278us) service_pool.cc:225] Handling call
1023 20:15:29.634892 (+1068422us) ts_tablet_manager.cc:1151] Deleting tablet 1a64a7c2e1f242e1b54243489622bae5
1023 20:15:29.634902 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.638938 (+ 4036us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.642958 (+ 4020us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.644103 (+ 1145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068231}
I20251023 20:15:29.644294 24363 rpcz_store.cc:276] 1023 20:15:28.591954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.592685 (+ 731us) service_pool.cc:225] Handling call
1023 20:15:29.644129 (+1051444us) ts_tablet_manager.cc:1151] Deleting tablet 39a62c97e0a3459e91c24c1c2f92389a
1023 20:15:29.644137 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.644214 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":445,"tablet-delete.queue_time_us":1051344}
I20251023 20:15:29.644675 23790 tablet_bootstrap.cc:492] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:29.644001 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
W20251023 20:15:29.644948 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
I20251023 20:15:29.644572 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 1a64a7c2e1f242e1b54243489622bae5 (table default.Table_d0d5b907c7a34f1aa8abea214c5c4cf0 [id=643c30acd68743bfb7c0ebd7b98199e7]) successfully deleted
I20251023 20:15:29.645045 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 186) took 1051 ms. Trace:
I20251023 20:15:29.645128 24363 rpcz_store.cc:276] 1023 20:15:28.593676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.593722 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:29.644953 (+1051231us) ts_tablet_manager.cc:1151] Deleting tablet d9f73477a06542c0b215ae86716cdc4e
1023 20:15:29.644959 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.645030 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051084}
W20251023 20:15:29.645460 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
I20251023 20:15:29.645579 24830 raft_consensus.cc:493] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.645715 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 187) took 1027 ms. Trace:
I20251023 20:15:29.645696 24830 raft_consensus.cc:515] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.645778 24363 rpcz_store.cc:276] 1023 20:15:28.618384 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.618436 (+ 52us) service_pool.cc:225] Handling call
1023 20:15:29.645629 (+1027193us) ts_tablet_manager.cc:1151] Deleting tablet f4778436aed64b53adb2bab54bcf263b
1023 20:15:29.645637 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.645700 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027009}
I20251023 20:15:29.646044 23790 tablet_bootstrap.cc:654] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.646116 23786 tablet_bootstrap.cc:492] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.646168 24363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 188) took 1013 ms. Trace:
I20251023 20:15:29.646225 23786 ts_tablet_manager.cc:1403] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20251023 20:15:29.646255 24830 leader_election.cc:290] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.646231 24363 rpcz_store.cc:276] 1023 20:15:28.632947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.633000 (+ 53us) service_pool.cc:225] Handling call
1023 20:15:29.646069 (+1013069us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:29.646077 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.646155 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1012867}
I20251023 20:15:29.646903 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.646953 23786 raft_consensus.cc:359] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.647096 23786 raft_consensus.cc:385] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.647145 23786 raft_consensus.cc:740] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.647257 24780 tablet_replica.cc:333] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.647221 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1cd0a61827c49fd99dfd054c9a292f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.647395 23664 raft_consensus.cc:2393] T d1cd0a61827c49fd99dfd054c9a292f8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:29.647316 23786 consensus_queue.cc:260] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.647419 24780 raft_consensus.cc:2243] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.647549 24780 raft_consensus.cc:2272] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.647758 23786 ts_tablet_manager.cc:1434] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:29.648030 23786 tablet_bootstrap.cc:492] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.648047 23470 leader_election.cc:304] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:29.649680 24780 ts_tablet_manager.cc:1916] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.650036 24830 raft_consensus.cc:3060] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:29.650342 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f4778436aed64b53adb2bab54bcf263b because the tablet was not found. No further retry: Not found: Tablet not found: f4778436aed64b53adb2bab54bcf263b
W20251023 20:15:29.650555 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
W20251023 20:15:29.650719 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6e1b4748c0244b258960b1025da47562 because the tablet was not found. No further retry: Not found: Tablet not found: 6e1b4748c0244b258960b1025da47562
W20251023 20:15:29.651003 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
I20251023 20:15:29.653456 24830 raft_consensus.cc:2749] T d1cd0a61827c49fd99dfd054c9a292f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:29.654973 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 8317a54a5f7b4afe8da9d336268efe43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8317a54a5f7b4afe8da9d336268efe43 already in progress: creating tablet
W20251023 20:15:29.655517 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 because the tablet was not found. No further retry: Not found: Tablet not found: a2dbb7997d0f4e6bac48cfb196f5ccb6
W20251023 20:15:29.655720 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a8ba7ab28dfa4ac38e0b58ba2115495c because the tablet was not found. No further retry: Not found: Tablet not found: a8ba7ab28dfa4ac38e0b58ba2115495c
W20251023 20:15:29.655925 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5346a221c78342359eb6e418bd12cd24 because the tablet was not found. No further retry: Not found: Tablet not found: 5346a221c78342359eb6e418bd12cd24
W20251023 20:15:29.656111 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 0438593fbbdf47cebfb64121acad104a because the tablet was not found. No further retry: Not found: Tablet not found: 0438593fbbdf47cebfb64121acad104a
I20251023 20:15:29.656191 24363 tablet_replica.cc:333] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:29.656342 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
I20251023 20:15:29.656383 24363 raft_consensus.cc:2243] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.656514 24363 raft_consensus.cc:2272] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.656904 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet edec01a63a3046d9ae5a372fc9068b52 because the tablet was not found. No further retry: Not found: Tablet not found: edec01a63a3046d9ae5a372fc9068b52
W20251023 20:15:29.657128 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
W20251023 20:15:29.657326 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
I20251023 20:15:29.657367 23787 tablet_bootstrap.cc:492] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.657477 23787 ts_tablet_manager.cc:1403] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.005s
W20251023 20:15:29.657516 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
W20251023 20:15:29.657733 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
W20251023 20:15:29.657939 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 0858358aabff47a69dc77f4669b225c5 because the tablet was not found. No further retry: Not found: Tablet not found: 0858358aabff47a69dc77f4669b225c5
W20251023 20:15:29.658130 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5964da70989b4e6cb77fe5d0a52ce9cf because the tablet was not found. No further retry: Not found: Tablet not found: 5964da70989b4e6cb77fe5d0a52ce9cf
I20251023 20:15:29.658461 23787 raft_consensus.cc:359] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.658674 23787 raft_consensus.cc:385] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.658759 23787 raft_consensus.cc:740] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.658993 23787 consensus_queue.cc:260] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.659526 23786 tablet_bootstrap.cc:654] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.659931 23787 ts_tablet_manager.cc:1434] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:29.660115 24363 ts_tablet_manager.cc:1916] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.660426 23787 tablet_bootstrap.cc:492] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.669829 23786 tablet_bootstrap.cc:492] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.669999 23786 ts_tablet_manager.cc:1403] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20251023 20:15:29.669999 24780 ts_tablet_manager.cc:1929] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.670091 24780 log.cc:1199] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/83dae34129794f49b43038d3b1636ea0
I20251023 20:15:29.670190 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:29.670384 24780 ts_tablet_manager.cc:1950] T 83dae34129794f49b43038d3b1636ea0 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.670506 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:29.670850 23786 raft_consensus.cc:359] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.671038 23786 raft_consensus.cc:385] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.671083 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 89) took 1104 ms. Trace:
I20251023 20:15:29.671134 23786 raft_consensus.cc:740] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.671162 24780 rpcz_store.cc:276] 1023 20:15:28.566816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.567067 (+ 251us) service_pool.cc:225] Handling call
1023 20:15:29.647217 (+1080150us) ts_tablet_manager.cc:1151] Deleting tablet 83dae34129794f49b43038d3b1636ea0
1023 20:15:29.647225 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.652724 (+ 5499us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.669899 (+ 17175us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.671064 (+ 1165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080025}
I20251023 20:15:29.671406 23786 consensus_queue.cc:260] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.671550 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 83dae34129794f49b43038d3b1636ea0 (table default.Table_52af8fa4ff964f16b6fe2b43e94dddd4 [id=872a29a7d5234fc98b6dd83be48499cd]) successfully deleted
I20251023 20:15:29.670935 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:29.672384 23786 ts_tablet_manager.cc:1434] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:29.672605 23787 tablet_bootstrap.cc:654] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.672669 23786 tablet_bootstrap.cc:492] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.674538 23786 tablet_bootstrap.cc:654] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.677601 24363 ts_tablet_manager.cc:1929] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.677711 24363 log.cc:1199] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/58ae6a6e58274b8baf952eaaefde7127
I20251023 20:15:29.672784 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:29.677992 24363 ts_tablet_manager.cc:1950] T 58ae6a6e58274b8baf952eaaefde7127 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.678018 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:29.678448 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:29.678767 23786 tablet_bootstrap.cc:492] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.678800 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:29.678908 23786 ts_tablet_manager.cc:1403] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251023 20:15:29.678952 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 58ae6a6e58274b8baf952eaaefde7127 (table default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]) successfully deleted
I20251023 20:15:29.679409 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
W20251023 20:15:29.679576 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
I20251023 20:15:29.679747 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:29.679713 23786 raft_consensus.cc:359] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.679888 23786 raft_consensus.cc:385] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.679979 23786 raft_consensus.cc:740] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:29.680117 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf because the tablet was not found. No further retry: Not found: Tablet not found: 1e8eb7603c3e46f1b15cdff7a7bbedcf
I20251023 20:15:29.680133 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:29.680192 23786 consensus_queue.cc:260] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.680486 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
W20251023 20:15:29.680644 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
W20251023 20:15:29.681149 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a21ec49194e24723b31fd9c86f1f85e4 because the tablet was not found. No further retry: Not found: Tablet not found: a21ec49194e24723b31fd9c86f1f85e4
I20251023 20:15:29.681293 24363 tablet_replica.cc:333] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:29.681363 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 40e0301fb5974ec5b6e8291230a736c2 because the tablet was not found. No further retry: Not found: Tablet not found: 40e0301fb5974ec5b6e8291230a736c2
I20251023 20:15:29.681456 24363 raft_consensus.cc:2243] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.681553 23786 ts_tablet_manager.cc:1434] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:29.681605 24363 raft_consensus.cc:2272] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.681821 23786 tablet_bootstrap.cc:492] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.682209 24363 ts_tablet_manager.cc:1916] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.683223 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 90) took 1115 ms. Trace:
I20251023 20:15:29.683324 24780 rpcz_store.cc:276] 1023 20:15:28.567353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.567622 (+ 269us) service_pool.cc:225] Handling call
1023 20:15:29.683084 (+1115462us) ts_tablet_manager.cc:1151] Deleting tablet 1e5720c96730425dafb550094cf936cd
1023 20:15:29.683097 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.683203 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115306}
W20251023 20:15:29.683802 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
I20251023 20:15:29.683926 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:29.684240 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:29.684528 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:29.685784 23786 tablet_bootstrap.cc:654] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.686995 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:29.687240 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:29.689240 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:29.689752 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:29.689973 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:29.690032 24363 ts_tablet_manager.cc:1929] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.690830 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:29.692206 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:29.694438 24363 log.cc:1199] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/25079532547d4175802b84877790662d
I20251023 20:15:29.695487 24363 ts_tablet_manager.cc:1950] T 25079532547d4175802b84877790662d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.695724 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:29.695907 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:29.696399 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:29.696733 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:29.692868 23787 tablet_bootstrap.cc:492] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.698551 24780 tablet_replica.cc:333] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.698758 24780 raft_consensus.cc:2243] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.698904 24780 raft_consensus.cc:2272] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.699429 24780 ts_tablet_manager.cc:1916] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.698069 23787 ts_tablet_manager.cc:1403] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20251023 20:15:29.702585 23787 raft_consensus.cc:359] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.702792 23787 raft_consensus.cc:385] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.702872 23787 raft_consensus.cc:740] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.700920 24363 tablet_replica.cc:333] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.703089 23787 consensus_queue.cc:260] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.703233 24363 raft_consensus.cc:2243] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.703611 23787 ts_tablet_manager.cc:1434] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:29.704120 23787 tablet_bootstrap.cc:492] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.704586 24363 raft_consensus.cc:2272] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.705818 24363 ts_tablet_manager.cc:1916] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.706104 23790 tablet_bootstrap.cc:492] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.706243 23790 ts_tablet_manager.cc:1403] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.000s
I20251023 20:15:29.706846 24780 ts_tablet_manager.cc:1929] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.706948 24780 log.cc:1199] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a9c5b69df4c04365b1ac1457fe8300cc
I20251023 20:15:29.706930 23790 raft_consensus.cc:359] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.707115 23790 raft_consensus.cc:385] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.707177 23790 raft_consensus.cc:740] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.707326 24780 ts_tablet_manager.cc:1950] T a9c5b69df4c04365b1ac1457fe8300cc P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.707355 23790 consensus_queue.cc:260] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.707778 23790 ts_tablet_manager.cc:1434] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251023 20:15:29.707993 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 91) took 1140 ms. Trace:
I20251023 20:15:29.708043 23790 tablet_bootstrap.cc:492] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.708073 24780 rpcz_store.cc:276] 1023 20:15:28.567925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.568169 (+ 244us) service_pool.cc:225] Handling call
1023 20:15:29.698487 (+1130318us) ts_tablet_manager.cc:1151] Deleting tablet a9c5b69df4c04365b1ac1457fe8300cc
1023 20:15:29.698496 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.702415 (+ 3919us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.706749 (+ 4334us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.707974 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1130188}
I20251023 20:15:29.709553 24780 tablet_replica.cc:333] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.709748 24780 raft_consensus.cc:2243] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.709888 23790 tablet_bootstrap.cc:654] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.710080 24780 raft_consensus.cc:2272] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.710631 24780 ts_tablet_manager.cc:1916] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.713294 23787 tablet_bootstrap.cc:654] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.717392 24363 ts_tablet_manager.cc:1929] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.719317 24780 ts_tablet_manager.cc:1929] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.719436 24780 log.cc:1199] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/7301d6121c9d4c8dbc642e895aa4e9f6
I20251023 20:15:29.719560 24363 log.cc:1199] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/994a555b24e4432a84601c54d422aa5d
I20251023 20:15:29.720424 24363 ts_tablet_manager.cc:1950] T 994a555b24e4432a84601c54d422aa5d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.722425 24780 ts_tablet_manager.cc:1950] T 7301d6121c9d4c8dbc642e895aa4e9f6 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.701148 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 25079532547d4175802b84877790662d (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]) successfully deleted
I20251023 20:15:29.709105 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a9c5b69df4c04365b1ac1457fe8300cc (table default.Table_8c497d9a1dcc41aa8466741225b4de85 [id=5746055cb51b4ee4b54b672987471cb1]) successfully deleted
W20251023 20:15:29.722955 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ac55e14745534a02a2f6a422cab6b149 because the tablet was not found. No further retry: Not found: Tablet not found: ac55e14745534a02a2f6a422cab6b149
W20251023 20:15:29.723209 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fe922c438a894a7aa04699b2d2897a28 because the tablet was not found. No further retry: Not found: Tablet not found: fe922c438a894a7aa04699b2d2897a28
W20251023 20:15:29.723392 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a27f192792cf41288d69ca15ab538472 because the tablet was not found. No further retry: Not found: Tablet not found: a27f192792cf41288d69ca15ab538472
I20251023 20:15:29.723455 23790 tablet_bootstrap.cc:492] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.723568 23790 ts_tablet_manager.cc:1403] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
W20251023 20:15:29.723615 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
W20251023 20:15:29.723799 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 15ced74d0dd74e5195ed98358957d0dd because the tablet was not found. No further retry: Not found: Tablet not found: 15ced74d0dd74e5195ed98358957d0dd
W20251023 20:15:29.723996 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a708cd011802458aa0e2991fd4b4a2d3 because the tablet was not found. No further retry: Not found: Tablet not found: a708cd011802458aa0e2991fd4b4a2d3
W20251023 20:15:29.724181 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
I20251023 20:15:29.724315 23790 raft_consensus.cc:359] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.724393 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
I20251023 20:15:29.724471 23790 raft_consensus.cc:385] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.724529 23790 raft_consensus.cc:740] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:29.724587 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2532c8567db14901b383f42e5f04b4c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2532c8567db14901b383f42e5f04b4c2
W20251023 20:15:29.724777 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d951134f6773410386bc57f7f90310c3 because the tablet was not found. No further retry: Not found: Tablet not found: d951134f6773410386bc57f7f90310c3
I20251023 20:15:29.724721 23790 consensus_queue.cc:260] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.724946 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]) successfully deleted
I20251023 20:15:29.725569 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 92) took 1157 ms. Trace:
I20251023 20:15:29.725885 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:29.725924 23790 ts_tablet_manager.cc:1434] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251023 20:15:29.726342 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 because the tablet was not found. No further retry: Not found: Tablet not found: 283166bbd8004ce3bca9f2ecbdbcbf48
I20251023 20:15:29.726744 23790 tablet_bootstrap.cc:492] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:29.726738 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 66a2f3adb5d347a0adc198cd1ef93304 because the tablet was not found. No further retry: Not found: Tablet not found: 66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:29.727166 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:29.727540 24363 tablet_replica.cc:333] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:29.727591 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 252b5480222f49439052c86a4115cc9b because the tablet was not found. No further retry: Not found: Tablet not found: 252b5480222f49439052c86a4115cc9b
I20251023 20:15:29.728596 24363 raft_consensus.cc:2243] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.728773 23790 tablet_bootstrap.cc:654] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.728889 24363 raft_consensus.cc:2272] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.727999 23787 tablet_bootstrap.cc:492] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.729497 23787 ts_tablet_manager.cc:1403] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20251023 20:15:29.730240 23787 raft_consensus.cc:359] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.732048 23787 raft_consensus.cc:385] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.732141 23787 raft_consensus.cc:740] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.729344 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:29.732609 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:29.732522 24780 rpcz_store.cc:276] 1023 20:15:28.568469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.568893 (+ 424us) service_pool.cc:225] Handling call
1023 20:15:29.709497 (+1140604us) ts_tablet_manager.cc:1151] Deleting tablet 7301d6121c9d4c8dbc642e895aa4e9f6
1023 20:15:29.709507 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.715820 (+ 6313us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.719216 (+ 3396us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.725547 (+ 6331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140468}
I20251023 20:15:29.733042 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 7301d6121c9d4c8dbc642e895aa4e9f6 (table default.Table_39094702f1ba4174a5eb05a40213e745 [id=2146c6f03d014ccdbc874c613400a8e5]) successfully deleted
I20251023 20:15:29.731968 24363 ts_tablet_manager.cc:1916] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.733907 24780 tablet_replica.cc:333] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.733919 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:29.734108 24780 raft_consensus.cc:2243] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.734090 23787 consensus_queue.cc:260] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.734259 24780 raft_consensus.cc:2272] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.734618 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:29.734790 24780 ts_tablet_manager.cc:1916] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.736163 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:29.736460 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:29.738024 23787 ts_tablet_manager.cc:1434] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:29.739985 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:29.741102 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:29.741488 23787 tablet_bootstrap.cc:492] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.743101 23787 tablet_bootstrap.cc:654] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.745106 24830 raft_consensus.cc:493] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.745258 24830 raft_consensus.cc:515] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.745911 24830 leader_election.cc:290] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.746726 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "994a555b24e4432a84601c54d422aa5d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.746860 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
W20251023 20:15:29.748430 23470 leader_election.cc:343] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 994a555b24e4432a84601c54d422aa5d
I20251023 20:15:29.748601 23470 leader_election.cc:304] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:29.749178 24830 raft_consensus.cc:2749] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.751041 24363 ts_tablet_manager.cc:1929] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.751886 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:29.753805 24363 log.cc:1199] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/419e41d7a37f4b888cb242b089c28790
I20251023 20:15:29.754056 23786 tablet_bootstrap.cc:492] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.754225 24363 ts_tablet_manager.cc:1950] T 419e41d7a37f4b888cb242b089c28790 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.754244 23786 ts_tablet_manager.cc:1403] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.001s
I20251023 20:15:29.755033 23786 raft_consensus.cc:359] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.755254 23786 raft_consensus.cc:385] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.755328 23786 raft_consensus.cc:740] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.755357 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 419e41d7a37f4b888cb242b089c28790 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:29.756565 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
W20251023 20:15:29.757122 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 951b2276d3324c2d999c8d7fb97767bc because the tablet was not found. No further retry: Not found: Tablet not found: 951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:29.757658 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
W20251023 20:15:29.758021 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
W20251023 20:15:29.758817 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 79e3d422a66f432789c0cf4a00e57af2 because the tablet was not found. No further retry: Not found: Tablet not found: 79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:29.759023 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:29.759294 23786 consensus_queue.cc:260] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.759990 23786 ts_tablet_manager.cc:1434] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:29.760764 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:29.761130 23786 tablet_bootstrap.cc:492] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.761891 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:29.763098 23786 tablet_bootstrap.cc:654] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.763127 24363 tablet_replica.cc:333] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:29.763336 24363 raft_consensus.cc:2243] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.763604 24363 raft_consensus.cc:2272] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.764256 24363 ts_tablet_manager.cc:1916] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.765584 23790 tablet_bootstrap.cc:492] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.766184 23790 ts_tablet_manager.cc:1403] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.006s
I20251023 20:15:29.767066 23790 raft_consensus.cc:359] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.767295 23790 raft_consensus.cc:385] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.767391 23790 raft_consensus.cc:740] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.767755 23790 consensus_queue.cc:260] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.768316 24780 ts_tablet_manager.cc:1929] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.768487 24780 log.cc:1199] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a8ba7ab28dfa4ac38e0b58ba2115495c
I20251023 20:15:29.769241 24780 ts_tablet_manager.cc:1950] T a8ba7ab28dfa4ac38e0b58ba2115495c P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.770133 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 93) took 1200 ms. Trace:
I20251023 20:15:29.770274 24780 rpcz_store.cc:276] 1023 20:15:28.569246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.569578 (+ 332us) service_pool.cc:225] Handling call
1023 20:15:29.733847 (+1164269us) ts_tablet_manager.cc:1151] Deleting tablet a8ba7ab28dfa4ac38e0b58ba2115495c
1023 20:15:29.733858 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.740402 (+ 6544us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.768202 (+ 27800us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.770110 (+ 1908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164130}
I20251023 20:15:29.771111 24780 tablet_replica.cc:333] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.771374 24780 raft_consensus.cc:2243] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.771770 24780 raft_consensus.cc:2272] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.773255 23786 tablet_bootstrap.cc:492] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.773468 23786 ts_tablet_manager.cc:1403] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20251023 20:15:29.774293 23786 raft_consensus.cc:359] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.774557 23786 raft_consensus.cc:385] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.774686 23786 raft_consensus.cc:740] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.774946 23786 consensus_queue.cc:260] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.775169 23790 ts_tablet_manager.cc:1434] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251023 20:15:29.778048 23786 ts_tablet_manager.cc:1434] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:29.778395 23786 tablet_bootstrap.cc:492] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.778873 23790 tablet_bootstrap.cc:492] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.779704 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:29.778527 24830 raft_consensus.cc:493] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.780640 24830 raft_consensus.cc:515] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.781278 24830 leader_election.cc:290] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.781291 24297 raft_consensus.cc:493] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.781404 24297 raft_consensus.cc:515] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.781853 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.781929 24297 leader_election.cc:290] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.782069 23664 raft_consensus.cc:2393] T b8e41ac34e84459581daef13914942b5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:29.782326 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8e41ac34e84459581daef13914942b5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:29.783134 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a04c65155ce048cb82b150cbe20be849" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.783298 23664 raft_consensus.cc:2468] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:29.783375 23786 tablet_bootstrap.cc:654] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.783735 23333 leader_election.cc:304] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:29.784116 24297 raft_consensus.cc:2804] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.784194 24297 raft_consensus.cc:493] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.784252 24297 raft_consensus.cc:3060] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.784276 23471 leader_election.cc:304] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:29.784353 24780 ts_tablet_manager.cc:1916] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.784878 24830 raft_consensus.cc:3060] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.784907 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a04c65155ce048cb82b150cbe20be849" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.786789 24297 raft_consensus.cc:515] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:29.786883 23333 consensus_peers.cc:597] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fa6a838b1bd409ba2a4e2074576ee23. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.787333 24297 leader_election.cc:290] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.787428 24830 raft_consensus.cc:2749] T b8e41ac34e84459581daef13914942b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:29.788031 23333 leader_election.cc:343] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.788146 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a04c65155ce048cb82b150cbe20be849" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.788467 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a04c65155ce048cb82b150cbe20be849" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.788633 23664 raft_consensus.cc:3060] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:29.788722 23333 leader_election.cc:343] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.770869 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a8ba7ab28dfa4ac38e0b58ba2115495c (table default.Table_4e18a968658f470d870bc932102ebf00 [id=ad634b38921149658d6212b02c421514]) successfully deleted
I20251023 20:15:29.789681 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
W20251023 20:15:29.790050 23333 consensus_peers.cc:597] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a708cd011802458aa0e2991fd4b4a2d3. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:29.791018 24780 ts_tablet_manager.cc:1929] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.791018 23664 raft_consensus.cc:2468] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.791103 24780 log.cc:1199] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/3fbad98d3ab24cc5b7726a92fba04cf8
I20251023 20:15:29.791453 24780 ts_tablet_manager.cc:1950] T 3fbad98d3ab24cc5b7726a92fba04cf8 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.791808 23214 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:29.791857 23333 leader_election.cc:304] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.792155 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 94) took 1222 ms. Trace:
I20251023 20:15:29.792189 24821 raft_consensus.cc:2804] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.792300 24821 raft_consensus.cc:697] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.792526 24821 consensus_queue.cc:237] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.795720 23790 tablet_bootstrap.cc:654] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.798622 23787 tablet_bootstrap.cc:492] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.798779 23787 ts_tablet_manager.cc:1403] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.002s
I20251023 20:15:29.798784 24780 rpcz_store.cc:276] 1023 20:15:28.569980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.570280 (+ 300us) service_pool.cc:225] Handling call
1023 20:15:29.771057 (+1200777us) ts_tablet_manager.cc:1151] Deleting tablet 3fbad98d3ab24cc5b7726a92fba04cf8
1023 20:15:29.771068 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.787596 (+ 16528us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.790922 (+ 3326us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.792138 (+ 1216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200635}
I20251023 20:15:29.799620 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 3fbad98d3ab24cc5b7726a92fba04cf8 (table default.Table_21f0d4fca62a4309816edd74c6b137bf [id=be3ef152e4fd4281b5caf942c8d404ab]) successfully deleted
I20251023 20:15:29.799693 24780 tablet_replica.cc:333] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.799888 24780 raft_consensus.cc:2243] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.799937 23787 raft_consensus.cc:359] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.800217 23787 raft_consensus.cc:385] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.800225 24780 raft_consensus.cc:2272] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.800311 23787 raft_consensus.cc:740] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.800580 23787 consensus_queue.cc:260] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.800814 24780 ts_tablet_manager.cc:1916] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.803962 23787 ts_tablet_manager.cc:1434] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:29.804913 23787 tablet_bootstrap.cc:492] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.811575 23787 tablet_bootstrap.cc:654] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.814625 24780 ts_tablet_manager.cc:1929] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.814767 24780 log.cc:1199] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/9d1ddde7e3134e23b3a9356523a93753
I20251023 20:15:29.814885 24363 ts_tablet_manager.cc:1929] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:29.815127 24780 ts_tablet_manager.cc:1950] T 9d1ddde7e3134e23b3a9356523a93753 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.815270 24363 log.cc:1199] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d2ae19cc80194363b8a9c96fbafb4ebe
I20251023 20:15:29.815799 24363 ts_tablet_manager.cc:1950] T d2ae19cc80194363b8a9c96fbafb4ebe P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:29.815972 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 95) took 1245 ms. Trace:
I20251023 20:15:29.816076 24780 rpcz_store.cc:276] 1023 20:15:28.570605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.570885 (+ 280us) service_pool.cc:225] Handling call
1023 20:15:29.799638 (+1228753us) ts_tablet_manager.cc:1151] Deleting tablet 9d1ddde7e3134e23b3a9356523a93753
1023 20:15:29.799650 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.807151 (+ 7501us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.814480 (+ 7329us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.815945 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228623}
I20251023 20:15:29.816614 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 9d1ddde7e3134e23b3a9356523a93753 (table default.Table_a1332462902c4ea5971afeb4c3688865 [id=f27defb723394b4db220b278b92e6ae8]) successfully deleted
I20251023 20:15:29.817149 24780 tablet_replica.cc:333] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.817409 24780 raft_consensus.cc:2243] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.817814 24780 pending_rounds.cc:70] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.817919 24780 pending_rounds.cc:77] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082158850879488 op_type: NO_OP noop_request { }
I20251023 20:15:29.818078 24780 raft_consensus.cc:2889] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.818365 24780 raft_consensus.cc:2272] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.818367 23786 tablet_bootstrap.cc:492] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.818598 23786 ts_tablet_manager.cc:1403] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.006s
I20251023 20:15:29.819054 24780 ts_tablet_manager.cc:1916] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.819428 23786 raft_consensus.cc:359] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.819670 23786 raft_consensus.cc:385] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.819752 23786 raft_consensus.cc:740] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.820008 23786 consensus_queue.cc:260] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.820626 23786 ts_tablet_manager.cc:1434] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:29.821105 23786 tablet_bootstrap.cc:492] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.822712 23786 tablet_bootstrap.cc:654] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.826367 24780 ts_tablet_manager.cc:1929] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.826591 24780 log.cc:1199] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/3a4c8dcbdfcf4a15b8a4194ea04b93a0
I20251023 20:15:29.826975 24780 ts_tablet_manager.cc:1950] T 3a4c8dcbdfcf4a15b8a4194ea04b93a0 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.827848 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 96) took 1256 ms. Trace:
I20251023 20:15:29.828006 24780 rpcz_store.cc:276] 1023 20:15:28.571173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.571431 (+ 258us) service_pool.cc:225] Handling call
1023 20:15:29.817064 (+1245633us) ts_tablet_manager.cc:1151] Deleting tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0
1023 20:15:29.817101 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.823001 (+ 5900us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.826259 (+ 3258us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.827822 (+ 1563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245491}
I20251023 20:15:29.828790 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 (table default.Table_e926815d67914fed915b184838f05b72 [id=265a9707ddea4ee1a19aac6033883822]) successfully deleted
I20251023 20:15:29.830783 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 97) took 1259 ms. Trace:
I20251023 20:15:29.831202 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d2ae19cc80194363b8a9c96fbafb4ebe (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]) successfully deleted
I20251023 20:15:29.830907 24780 rpcz_store.cc:276] 1023 20:15:28.571709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.571952 (+ 243us) service_pool.cc:225] Handling call
1023 20:15:29.830680 (+1258728us) ts_tablet_manager.cc:1151] Deleting tablet 33b5efb30b5c491c8eafe3ac25688843
1023 20:15:29.830690 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.830763 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258605}
W20251023 20:15:29.831851 23205 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 33b5efb30b5c491c8eafe3ac25688843 because the tablet was not found. No further retry: Not found: Tablet not found: 33b5efb30b5c491c8eafe3ac25688843
I20251023 20:15:29.832238 24780 tablet_replica.cc:333] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.832484 24780 raft_consensus.cc:2243] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.832556 24658 raft_consensus.cc:493] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.832711 24780 raft_consensus.cc:2272] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.832686 24658 raft_consensus.cc:515] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.833261 24780 ts_tablet_manager.cc:1916] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.833318 24658 leader_election.cc:290] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.833881 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bfbf0acdc84f2f8fe553694ee9897c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:29.834584 23604 leader_election.cc:343] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.834648 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bfbf0acdc84f2f8fe553694ee9897c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:29.835234 23605 leader_election.cc:343] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.835342 23605 leader_election.cc:304] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.835798 24658 raft_consensus.cc:2749] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.841996 24780 ts_tablet_manager.cc:1929] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.842132 24780 log.cc:1199] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/83da09ee0d10435ba89a19dc378d34eb
I20251023 20:15:29.842499 24780 ts_tablet_manager.cc:1950] T 83da09ee0d10435ba89a19dc378d34eb P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.843318 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 98) took 1271 ms. Trace:
I20251023 20:15:29.843423 24780 rpcz_store.cc:276] 1023 20:15:28.572216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.572449 (+ 233us) service_pool.cc:225] Handling call
1023 20:15:29.832180 (+1259731us) ts_tablet_manager.cc:1151] Deleting tablet 83da09ee0d10435ba89a19dc378d34eb
1023 20:15:29.832192 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.836344 (+ 4152us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.841862 (+ 5518us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.843294 (+ 1432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1259605}
I20251023 20:15:29.843969 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 83da09ee0d10435ba89a19dc378d34eb (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]) successfully deleted
I20251023 20:15:29.844496 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 99) took 1271 ms. Trace:
I20251023 20:15:29.844640 24780 rpcz_store.cc:276] 1023 20:15:28.572706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.572927 (+ 221us) service_pool.cc:225] Handling call
1023 20:15:29.844357 (+1271430us) ts_tablet_manager.cc:1151] Deleting tablet 01fa5f9c700b458ea147373d78551573
1023 20:15:29.844368 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.844476 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1271312}
W20251023 20:15:29.845297 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
I20251023 20:15:29.845733 24780 tablet_replica.cc:333] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.846001 24780 raft_consensus.cc:2243] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.846218 24780 raft_consensus.cc:2272] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.846889 24780 ts_tablet_manager.cc:1916] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.853332 24780 ts_tablet_manager.cc:1929] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.853567 24780 log.cc:1199] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/523d72012c364457b9a2a299f2f789a1
I20251023 20:15:29.853986 24780 ts_tablet_manager.cc:1950] T 523d72012c364457b9a2a299f2f789a1 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.854821 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 100) took 1281 ms. Trace:
I20251023 20:15:29.854981 24780 rpcz_store.cc:276] 1023 20:15:28.573222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.573473 (+ 251us) service_pool.cc:225] Handling call
1023 20:15:29.845681 (+1272208us) ts_tablet_manager.cc:1151] Deleting tablet 523d72012c364457b9a2a299f2f789a1
1023 20:15:29.845693 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.850221 (+ 4528us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.853223 (+ 3002us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.854798 (+ 1575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1272076}
I20251023 20:15:29.855611 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 523d72012c364457b9a2a299f2f789a1 (table default.Table_7e026bfaccc4482dba743310ed0adb4c [id=7fdd0d3b02734ea89c0fce5e98fca82e]) successfully deleted
I20251023 20:15:29.855978 24780 tablet_replica.cc:333] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.856195 24780 raft_consensus.cc:2243] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.856578 24780 pending_rounds.cc:70] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.856674 24780 pending_rounds.cc:77] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082164471873536 op_type: NO_OP noop_request { }
I20251023 20:15:29.856817 24780 raft_consensus.cc:2889] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.856943 24780 raft_consensus.cc:2272] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.857632 24780 ts_tablet_manager.cc:1916] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.857870 23790 tablet_bootstrap.cc:492] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.858078 23790 ts_tablet_manager.cc:1403] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.001s
I20251023 20:15:29.858901 23790 raft_consensus.cc:359] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.859153 23790 raft_consensus.cc:385] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.859241 23790 raft_consensus.cc:740] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.859527 23790 consensus_queue.cc:260] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.860139 23790 ts_tablet_manager.cc:1434] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:29.860476 23790 tablet_bootstrap.cc:492] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.862278 23790 tablet_bootstrap.cc:654] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.864411 23787 tablet_bootstrap.cc:492] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.864569 23787 ts_tablet_manager.cc:1403] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.012s
I20251023 20:15:29.865298 23787 raft_consensus.cc:359] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.865471 23787 raft_consensus.cc:385] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.865530 23787 raft_consensus.cc:740] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.865806 24780 ts_tablet_manager.cc:1929] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.866146 24780 log.cc:1199] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/af899c01b4af46d39d27f984cc3d5940
I20251023 20:15:29.866528 24780 ts_tablet_manager.cc:1950] T af899c01b4af46d39d27f984cc3d5940 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.867314 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 101) took 1293 ms. Trace:
I20251023 20:15:29.867406 24780 rpcz_store.cc:276] 1023 20:15:28.573759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.574007 (+ 248us) service_pool.cc:225] Handling call
1023 20:15:29.855932 (+1281925us) ts_tablet_manager.cc:1151] Deleting tablet af899c01b4af46d39d27f984cc3d5940
1023 20:15:29.855942 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.862665 (+ 6723us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.865701 (+ 3036us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.867280 (+ 1579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1281805}
I20251023 20:15:29.868152 24780 tablet_replica.cc:333] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.868348 24780 raft_consensus.cc:2243] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.868700 24780 raft_consensus.cc:2272] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.869267 24780 ts_tablet_manager.cc:1916] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.870031 23787 consensus_queue.cc:260] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.875581 24780 ts_tablet_manager.cc:1929] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.875638 23787 ts_tablet_manager.cc:1434] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251023 20:15:29.875726 24780 log.cc:1199] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/5e73ae95c73945f081c298f49d47c904
I20251023 20:15:29.876055 24780 ts_tablet_manager.cc:1950] T 5e73ae95c73945f081c298f49d47c904 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.876027 24979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.127.189:33509 every 8 connection(s)
I20251023 20:15:29.876797 24423 rpc_server.cc:307] RPC server started. Bound to: 127.22.127.189:33509
I20251023 20:15:29.876808 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 102) took 1302 ms. Trace:
I20251023 20:15:29.876878 24780 rpcz_store.cc:276] 1023 20:15:28.574244 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.574483 (+ 239us) service_pool.cc:225] Handling call
1023 20:15:29.868100 (+1293617us) ts_tablet_manager.cc:1151] Deleting tablet 5e73ae95c73945f081c298f49d47c904
1023 20:15:29.868111 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.872420 (+ 4309us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.875460 (+ 3040us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.876786 (+ 1326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293466}
I20251023 20:15:29.877206 23786 tablet_bootstrap.cc:492] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.877375 24780 tablet_replica.cc:333] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.877415 23786 ts_tablet_manager.cc:1403] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.004s
I20251023 20:15:29.877570 24780 raft_consensus.cc:2243] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.877718 24780 raft_consensus.cc:2272] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.878214 24780 ts_tablet_manager.cc:1916] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.878443 23786 raft_consensus.cc:359] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.878701 23786 raft_consensus.cc:385] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.878861 23786 raft_consensus.cc:740] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.878993 24423 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-1/data/info.pb
I20251023 20:15:29.879293 23786 consensus_queue.cc:260] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.881760 23786 ts_tablet_manager.cc:1434] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:29.882156 23786 tablet_bootstrap.cc:492] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.885252 23038 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu as pid 24423
I20251023 20:15:29.889587 23786 tablet_bootstrap.cc:654] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.867940 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]) successfully deleted
I20251023 20:15:29.891091 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 5e73ae95c73945f081c298f49d47c904 (table default.Table_8d8ada20a7bb43eb9a534aedca23cfd2 [id=6fb3e731496a466a918ab84247724a22]) successfully deleted
I20251023 20:15:29.891291 23790 tablet_bootstrap.cc:492] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.891474 23790 ts_tablet_manager.cc:1403] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20251023 20:15:29.892273 23790 raft_consensus.cc:359] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.892514 23790 raft_consensus.cc:385] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.892604 23790 raft_consensus.cc:740] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.892865 23790 consensus_queue.cc:260] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.895185 23790 ts_tablet_manager.cc:1434] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:29.895509 23790 tablet_bootstrap.cc:492] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.897195 23790 tablet_bootstrap.cc:654] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.900691 24830 raft_consensus.cc:493] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.900885 24830 raft_consensus.cc:515] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:29.901496 24830 leader_election.cc:290] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.903680 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
W20251023 20:15:29.904328 23470 leader_election.cc:343] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: f103bece2fa240c19c23ac51046c13ed
I20251023 20:15:29.904592 23790 tablet_bootstrap.cc:492] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.905171 23790 ts_tablet_manager.cc:1403] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
W20251023 20:15:29.905171 23471 leader_election.cc:343] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: f103bece2fa240c19c23ac51046c13ed
I20251023 20:15:29.905290 23471 leader_election.cc:304] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:29.905740 24830 raft_consensus.cc:2749] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:29.906116 23790 raft_consensus.cc:359] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.907171 23790 raft_consensus.cc:385] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.907373 23790 raft_consensus.cc:740] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.907774 23790 consensus_queue.cc:260] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.907845 24658 raft_consensus.cc:493] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.907974 24658 raft_consensus.cc:515] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.904937 24780 ts_tablet_manager.cc:1929] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:29.907385 24670 raft_consensus.cc:493] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.908355 24670 raft_consensus.cc:515] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.908561 24658 leader_election.cc:290] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.908890 24670 leader_election.cc:290] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.909052 23790 ts_tablet_manager.cc:1434] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:29.909673 24780 log.cc:1199] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/f908f086946949829cac15418a584730
I20251023 20:15:29.909865 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.909724 23790 tablet_bootstrap.cc:492] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.910023 24780 ts_tablet_manager.cc:1950] T f908f086946949829cac15418a584730 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.910274 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 103) took 1335 ms. Trace:
I20251023 20:15:29.910259 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:29.910337 24780 rpcz_store.cc:276] 1023 20:15:28.574807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.575082 (+ 275us) service_pool.cc:225] Handling call
1023 20:15:29.877329 (+1302247us) ts_tablet_manager.cc:1151] Deleting tablet f908f086946949829cac15418a584730
1023 20:15:29.877338 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.883896 (+ 6558us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.904815 (+ 20919us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.910259 (+ 5444us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1302105}
I20251023 20:15:29.910784 24780 tablet_replica.cc:333] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.910796 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet f908f086946949829cac15418a584730 (table default.Table_a1c9b5357e504ba2a116c58ff86ad13c [id=25a11be83f484fa6abecb21ce1145e1e]) successfully deleted
I20251023 20:15:29.911005 24780 raft_consensus.cc:2243] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:29.911176 24780 raft_consensus.cc:2272] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:29.911451 23604 leader_election.cc:343] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.911715 24780 ts_tablet_manager.cc:1916] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.911787 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:29.911947 23664 raft_consensus.cc:2468] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:29.912104 23790 tablet_bootstrap.cc:654] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.912987 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:29.913540 23605 leader_election.cc:343] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.913647 23605 leader_election.cc:304] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.914068 24658 raft_consensus.cc:2749] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:29.914494 23333 leader_election.cc:343] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.914687 23333 leader_election.cc:304] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.914921 24670 raft_consensus.cc:2804] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:29.914997 24670 raft_consensus.cc:493] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:29.915067 24670 raft_consensus.cc:3060] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.917701 24670 raft_consensus.cc:515] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.918322 24670 leader_election.cc:290] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:29.918794 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:29.918962 23664 raft_consensus.cc:3060] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:29.921597 23787 tablet_bootstrap.cc:492] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.923131 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:29.923249 23787 tablet_bootstrap.cc:654] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:29.923713 23333 leader_election.cc:343] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.924474 23664 raft_consensus.cc:2468] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:29.926752 23333 leader_election.cc:304] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:29.927266 24670 raft_consensus.cc:2804] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:29.927386 24670 raft_consensus.cc:697] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:29.927625 24670 consensus_queue.cc:237] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.932130 24780 ts_tablet_manager.cc:1929] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.932265 24780 log.cc:1199] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/cff337a6e728489b99fa8c5d4dfece2b
I20251023 20:15:29.932632 24780 ts_tablet_manager.cc:1950] T cff337a6e728489b99fa8c5d4dfece2b P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.932942 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 104) took 1357 ms. Trace:
I20251023 20:15:29.935452 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:29.935671 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:29.935832 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:29.936059 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:29.936592 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:29.936839 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:29.933024 24780 rpcz_store.cc:276] 1023 20:15:28.575577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.575897 (+ 320us) service_pool.cc:225] Handling call
1023 20:15:29.910738 (+1334841us) ts_tablet_manager.cc:1151] Deleting tablet cff337a6e728489b99fa8c5d4dfece2b
1023 20:15:29.910747 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.914786 (+ 4039us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.932012 (+ 17226us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.932926 (+ 914us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334711}
I20251023 20:15:29.937827 24780 tablet_replica.cc:333] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.938017 24780 raft_consensus.cc:2243] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.938321 23790 tablet_bootstrap.cc:492] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:29.938522 23790 ts_tablet_manager.cc:1403] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.004s
I20251023 20:15:29.939314 23790 raft_consensus.cc:359] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.939554 23790 raft_consensus.cc:385] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.939652 23790 raft_consensus.cc:740] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.939888 23790 consensus_queue.cc:260] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.940439 23790 ts_tablet_manager.cc:1434] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:29.941807 24981 sys_catalog.cc:263] Verifying existing consensus state
I20251023 20:15:29.938360 24780 raft_consensus.cc:2272] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.942483 24780 ts_tablet_manager.cc:1916] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.945546 23790 tablet_bootstrap.cc:492] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:29.946514 24981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Bootstrap starting.
I20251023 20:15:29.949343 24780 ts_tablet_manager.cc:1929] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:29.950757 23786 tablet_bootstrap.cc:492] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:29.950933 23786 ts_tablet_manager.cc:1403] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.002s
I20251023 20:15:29.951463 23787 tablet_bootstrap.cc:492] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.951628 23786 raft_consensus.cc:359] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.951828 23786 raft_consensus.cc:385] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.951905 23786 raft_consensus.cc:740] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.952131 23787 ts_tablet_manager.cc:1403] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20251023 20:15:29.952126 23786 consensus_queue.cc:260] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.961580 23786 ts_tablet_manager.cc:1434] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251023 20:15:29.961920 23786 tablet_bootstrap.cc:492] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:29.968061 23787 raft_consensus.cc:359] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.968408 24780 log.cc:1199] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/52bc660b9da54c5d984381a267e1efa2
I20251023 20:15:29.968567 23787 raft_consensus.cc:385] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.969022 24780 ts_tablet_manager.cc:1950] T 52bc660b9da54c5d984381a267e1efa2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:29.969158 23787 raft_consensus.cc:740] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.969619 23787 consensus_queue.cc:260] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.954010 23790 tablet_bootstrap.cc:654] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.970357 23787 ts_tablet_manager.cc:1434] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251023 20:15:29.971143 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 105) took 1394 ms. Trace:
I20251023 20:15:29.971501 23787 tablet_bootstrap.cc:492] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:29.973122 23787 tablet_bootstrap.cc:654] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:29.973181 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet cff337a6e728489b99fa8c5d4dfece2b (table default.Table_ca4eab8d09574f7abb17e56482816adf [id=1abfa2f299624b41be063c4934d88e64]) successfully deleted
I20251023 20:15:29.973223 24780 rpcz_store.cc:276] 1023 20:15:28.576182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.576444 (+ 262us) service_pool.cc:225] Handling call
1023 20:15:29.937770 (+1361326us) ts_tablet_manager.cc:1151] Deleting tablet 52bc660b9da54c5d984381a267e1efa2
1023 20:15:29.937778 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.945445 (+ 7667us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.949255 (+ 3810us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:29.971122 (+ 21867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361132}
I20251023 20:15:29.973563 24780 tablet_replica.cc:333] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:29.973755 24780 raft_consensus.cc:2243] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:29.973965 24780 pending_rounds.cc:70] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:29.974014 24780 pending_rounds.cc:77] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082168438378496 op_type: NO_OP noop_request { }
I20251023 20:15:29.974097 24780 raft_consensus.cc:2889] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:29.974170 24780 raft_consensus.cc:2272] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:29.974892 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 52bc660b9da54c5d984381a267e1efa2 (table default.Table_0cd3fe0039de476c813272d09a9ba564 [id=161b57ca7cf148899ea058eab98b780d]) successfully deleted
I20251023 20:15:29.975564 24780 ts_tablet_manager.cc:1916] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:29.986824 23787 tablet_bootstrap.cc:492] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:29.986989 23787 ts_tablet_manager.cc:1403] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251023 20:15:29.989871 23787 raft_consensus.cc:359] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.990063 23787 raft_consensus.cc:385] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:29.990119 23787 raft_consensus.cc:740] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:29.990378 23787 consensus_queue.cc:260] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.990790 24658 raft_consensus.cc:493] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:29.990907 24658 raft_consensus.cc:515] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:29.991496 24658 leader_election.cc:290] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:29.992411 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:29.993877 23605 leader_election.cc:343] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:29.973140 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:29.995425 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:29.995949 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:29.996131 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:29.999410 23787 ts_tablet_manager.cc:1434] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20251023 20:15:30.002768 23790 tablet_bootstrap.cc:492] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.002939 23790 ts_tablet_manager.cc:1403] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.002s
I20251023 20:15:30.007486 23787 tablet_bootstrap.cc:492] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.008631 23790 raft_consensus.cc:359] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.008807 23790 raft_consensus.cc:385] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.008854 23790 raft_consensus.cc:740] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.009047 23790 consensus_queue.cc:260] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.009609 23790 ts_tablet_manager.cc:1434] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:30.009884 23790 tablet_bootstrap.cc:492] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.011535 24780 ts_tablet_manager.cc:1929] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.012867 23787 tablet_bootstrap.cc:654] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.013020 24780 log.cc:1199] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/46f700ea3db04a4bad539c943e850a44
I20251023 20:15:30.013142 24821 raft_consensus.cc:493] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.013259 24821 raft_consensus.cc:515] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.013451 24780 ts_tablet_manager.cc:1950] T 46f700ea3db04a4bad539c943e850a44 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.013787 24821 leader_election.cc:290] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.014230 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 106) took 1437 ms. Trace:
I20251023 20:15:30.014314 24780 rpcz_store.cc:276] 1023 20:15:28.576758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.577015 (+ 257us) service_pool.cc:225] Handling call
1023 20:15:29.973500 (+1396485us) ts_tablet_manager.cc:1151] Deleting tablet 46f700ea3db04a4bad539c943e850a44
1023 20:15:29.973510 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:29.979580 (+ 6070us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.011405 (+ 31825us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.014207 (+ 2802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1396287}
I20251023 20:15:30.014654 24780 tablet_replica.cc:333] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:30.014834 24780 raft_consensus.cc:2243] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:30.014971 23333 leader_election.cc:343] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 419e41d7a37f4b888cb242b089c28790
I20251023 20:15:30.015067 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 46f700ea3db04a4bad539c943e850a44 (table default.Table_aba84ed1ff684cc690f31c5615ec1700 [id=0b966d7b9d4d4ad1bdf1c84d63c6be5a]) successfully deleted
I20251023 20:15:30.015174 24780 raft_consensus.cc:2272] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.015733 24780 ts_tablet_manager.cc:1916] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.018080 23790 tablet_bootstrap.cc:654] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.020268 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:30.020550 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:30.021983 23217 catalog_manager.cc:5649] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:30.022431 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:30.022652 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:30.022920 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:30.024269 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:30.024775 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:30.024817 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.024925 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:30.025121 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:30.025394 23786 tablet_bootstrap.cc:654] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:30.025367 23604 leader_election.cc:343] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.025405 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:30.025478 23604 leader_election.cc:304] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.025561 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:30.025816 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:30.025871 24658 raft_consensus.cc:2749] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.026135 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:30.026276 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:30.026558 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:30.026548 24830 raft_consensus.cc:493] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.026844 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:30.027061 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:30.027305 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:30.027256 24830 raft_consensus.cc:515] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:30.027866 24830 leader_election.cc:290] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.027948 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:30.028203 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:30.028463 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:30.029584 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:30.029718 24658 raft_consensus.cc:493] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.029837 24658 raft_consensus.cc:515] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.030350 24658 leader_election.cc:290] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:30.031167 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c58a406d564b048fb16a85aa0b754b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:30.031898 23470 leader_election.cc:343] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a636a4cdeb404e9f806d8e166ea57938
I20251023 20:15:30.032029 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:30.032343 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:30.032359 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c58a406d564b048fb16a85aa0b754b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.032570 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
W20251023 20:15:30.033794 23605 leader_election.cc:343] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.034205 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
W20251023 20:15:30.034626 23604 leader_election.cc:343] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.034730 23604 leader_election.cc:304] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.035360 24658 raft_consensus.cc:2749] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.035912 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:30.036299 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:30.036850 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:30.037133 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:30.037453 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:30.037798 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:30.037953 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:30.038084 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:30.038348 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:30.038725 23786 tablet_bootstrap.cc:492] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.038758 23217 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:30.038884 23786 ts_tablet_manager.cc:1403] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.001s
I20251023 20:15:30.039634 23786 raft_consensus.cc:359] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.039824 23786 raft_consensus.cc:385] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.039912 23786 raft_consensus.cc:740] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.040123 23786 consensus_queue.cc:260] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.040714 23786 ts_tablet_manager.cc:1434] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:30.041041 23786 tablet_bootstrap.cc:492] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.043044 23787 tablet_bootstrap.cc:492] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.043177 23787 ts_tablet_manager.cc:1403] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.002s
I20251023 20:15:30.043393 23786 tablet_bootstrap.cc:654] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.044518 23787 raft_consensus.cc:359] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.044703 23787 raft_consensus.cc:385] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.044765 23787 raft_consensus.cc:740] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.044787 23446 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:30.045007 23787 consensus_queue.cc:260] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.045511 23787 ts_tablet_manager.cc:1434] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251023 20:15:30.046496 24830 raft_consensus.cc:493] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.046633 24830 raft_consensus.cc:515] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:30.047676 24830 leader_election.cc:290] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.047716 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.047923 23670 raft_consensus.cc:2393] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
W20251023 20:15:30.050233 23333 consensus_peers.cc:597] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2ae19cc80194363b8a9c96fbafb4ebe. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.057350 23787 tablet_bootstrap.cc:492] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.064814 23787 tablet_bootstrap.cc:654] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.065184 24658 raft_consensus.cc:493] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.065322 24658 raft_consensus.cc:515] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.065959 24658 leader_election.cc:290] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.067315 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb475e303d07491bbc19dc3f628bceb7" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.067804 23333 consensus_peers.cc:597] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 925e53d9ea6f4c7baa38551eb917828d. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.068657 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb475e303d07491bbc19dc3f628bceb7" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:30.069458 23604 leader_election.cc:343] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.069609 24780 ts_tablet_manager.cc:1929] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.069716 24780 log.cc:1199] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/4ea63540149c4c02abb8ae51209060a3
W20251023 20:15:30.069723 23605 leader_election.cc:343] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.069864 23605 leader_election.cc:304] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.070088 24780 ts_tablet_manager.cc:1950] T 4ea63540149c4c02abb8ae51209060a3 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.070410 24297 consensus_queue.cc:1048] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.070833 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 107) took 1493 ms. Trace:
I20251023 20:15:30.070912 24780 rpcz_store.cc:276] 1023 20:15:28.577330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.577649 (+ 319us) service_pool.cc:225] Handling call
1023 20:15:30.014588 (+1436939us) ts_tablet_manager.cc:1151] Deleting tablet 4ea63540149c4c02abb8ae51209060a3
1023 20:15:30.014598 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.040786 (+ 26188us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.069501 (+ 28715us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.070813 (+ 1312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436761}
I20251023 20:15:30.071370 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 4ea63540149c4c02abb8ae51209060a3 (table default.Table_e8cec02c09a44acab531a2f71ebb0a73 [id=41b1a0d78a924f2782bcfd5695f835d1]) successfully deleted
I20251023 20:15:30.072034 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 109) took 1493 ms. Trace:
I20251023 20:15:30.072106 24780 rpcz_store.cc:276] 1023 20:15:28.578714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.578955 (+ 241us) service_pool.cc:225] Handling call
1023 20:15:30.071914 (+1492959us) ts_tablet_manager.cc:1151] Deleting tablet e194d03a6aa946fbb2c79d91b196440b
1023 20:15:30.071924 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.072016 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492740}
W20251023 20:15:30.072571 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
I20251023 20:15:30.073258 24658 raft_consensus.cc:2749] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.077667 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 110) took 1498 ms. Trace:
I20251023 20:15:30.077694 24821 raft_consensus.cc:493] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea6853efbbcc484eb7529c7e9b37a47a)
I20251023 20:15:30.077823 24821 raft_consensus.cc:515] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.077765 24780 rpcz_store.cc:276] 1023 20:15:28.579990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.580104 (+ 114us) service_pool.cc:225] Handling call
1023 20:15:30.077540 (+1497436us) ts_tablet_manager.cc:1151] Deleting tablet 7e36bb5e61d34b35a373bd6a51a7ac5e
1023 20:15:30.077551 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.077646 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497301}
I20251023 20:15:30.078375 24821 leader_election.cc:290] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.078518 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 111) took 1498 ms. Trace:
I20251023 20:15:30.078598 24780 rpcz_store.cc:276] 1023 20:15:28.580343 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.580573 (+ 230us) service_pool.cc:225] Handling call
1023 20:15:30.078413 (+1497840us) ts_tablet_manager.cc:1151] Deleting tablet 76380b341a304d6ba31648fbd1548f05
1023 20:15:30.078422 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.078503 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497722}
I20251023 20:15:30.078840 24780 tablet_replica.cc:333] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:30.079015 24780 raft_consensus.cc:2243] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.079156 24780 raft_consensus.cc:2272] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:30.079152 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
W20251023 20:15:30.079430 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
W20251023 20:15:30.079855 23333 leader_election.cc:343] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:30.080165 24821 raft_consensus.cc:493] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.080281 24821 raft_consensus.cc:515] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.080854 24821 leader_election.cc:290] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.081393 23787 tablet_bootstrap.cc:492] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:30.081543 23333 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:30.081573 23787 ts_tablet_manager.cc:1403] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
I20251023 20:15:30.082388 23787 raft_consensus.cc:359] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.082630 23787 raft_consensus.cc:385] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.082724 23787 raft_consensus.cc:740] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.082906 24981 log.cc:826] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Log is configured to *not* fsync() on all Append() calls
I20251023 20:15:30.083011 23787 consensus_queue.cc:260] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.083442 23786 tablet_bootstrap.cc:492] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.083635 23786 ts_tablet_manager.cc:1403] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.005s
I20251023 20:15:30.084297 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:30.085016 23786 raft_consensus.cc:359] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.085404 23786 raft_consensus.cc:385] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.085484 23786 raft_consensus.cc:740] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.085731 23786 consensus_queue.cc:260] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.086236 23786 ts_tablet_manager.cc:1434] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:30.088433 23787 ts_tablet_manager.cc:1434] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251023 20:15:30.089946 23787 tablet_bootstrap.cc:492] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.095702 23787 tablet_bootstrap.cc:654] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.095906 24780 ts_tablet_manager.cc:1916] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:30.102418 23333 consensus_peers.cc:597] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 925e53d9ea6f4c7baa38551eb917828d. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.104312 24780 ts_tablet_manager.cc:1929] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:30.105020 23333 consensus_peers.cc:597] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15ced74d0dd74e5195ed98358957d0dd. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.106027 23786 tablet_bootstrap.cc:492] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.106220 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "744d31fccbbe4fa59c56e24136e3b2b9" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.106703 24780 log.cc:1199] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/4beebee06e5341a29c65d0c3cf985ef4
I20251023 20:15:30.106691 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "419e41d7a37f4b888cb242b089c28790" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.107143 24780 ts_tablet_manager.cc:1950] T 4beebee06e5341a29c65d0c3cf985ef4 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.107872 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 112) took 1527 ms. Trace:
I20251023 20:15:30.107949 24780 rpcz_store.cc:276] 1023 20:15:28.580790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.581016 (+ 226us) service_pool.cc:225] Handling call
1023 20:15:30.078798 (+1497782us) ts_tablet_manager.cc:1151] Deleting tablet 4beebee06e5341a29c65d0c3cf985ef4
1023 20:15:30.078806 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.098971 (+ 20165us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.104202 (+ 5231us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.107853 (+ 3651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497647}
I20251023 20:15:30.108029 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a54470d088748b0be22ab9db5a25f87" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.108335 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 4beebee06e5341a29c65d0c3cf985ef4 (table default.Table_3220c87e07554676ba34368830409e89 [id=85bda6747ffe49f1b5a123f41013c426]) successfully deleted
I20251023 20:15:30.109787 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 113) took 1528 ms. Trace:
I20251023 20:15:30.109869 24780 rpcz_store.cc:276] 1023 20:15:28.581254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.581477 (+ 223us) service_pool.cc:225] Handling call
1023 20:15:30.109664 (+1528187us) ts_tablet_manager.cc:1151] Deleting tablet 8317a54a5f7b4afe8da9d336268efe43
1023 20:15:30.109673 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.109762 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528036}
W20251023 20:15:30.110265 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 8317a54a5f7b4afe8da9d336268efe43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8317a54a5f7b4afe8da9d336268efe43 already in progress: creating tablet
I20251023 20:15:30.111317 23787 tablet_bootstrap.cc:492] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.111452 23787 ts_tablet_manager.cc:1403] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.002s
I20251023 20:15:30.112195 23787 raft_consensus.cc:359] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.112358 23787 raft_consensus.cc:385] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.112409 23787 raft_consensus.cc:740] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.112638 23786 tablet_bootstrap.cc:654] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.112625 23787 consensus_queue.cc:260] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:30.116425 23333 leader_election.cc:343] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.116552 23333 leader_election.cc:304] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.116885 24821 raft_consensus.cc:2749] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.118897 23790 tablet_bootstrap.cc:492] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.119040 23790 ts_tablet_manager.cc:1403] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.003s
I20251023 20:15:30.119807 23790 raft_consensus.cc:359] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.119901 23787 ts_tablet_manager.cc:1434] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:30.119966 23790 raft_consensus.cc:385] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.120016 23790 raft_consensus.cc:740] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.120252 23790 consensus_queue.cc:260] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.120687 23790 ts_tablet_manager.cc:1434] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:30.120946 23790 tablet_bootstrap.cc:492] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.121429 23787 tablet_bootstrap.cc:492] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.122480 23790 tablet_bootstrap.cc:654] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.122879 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 114) took 1541 ms. Trace:
I20251023 20:15:30.123358 23787 tablet_bootstrap.cc:654] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.123457 24780 rpcz_store.cc:276] 1023 20:15:28.581755 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.582015 (+ 260us) service_pool.cc:225] Handling call
1023 20:15:30.122730 (+1540715us) ts_tablet_manager.cc:1151] Deleting tablet ca11ddd5d97544cf916b87fd3dff219d
1023 20:15:30.122743 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.122858 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540596}
I20251023 20:15:30.124661 24780 tablet_replica.cc:333] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
W20251023 20:15:30.124650 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
I20251023 20:15:30.124976 24780 raft_consensus.cc:2243] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.125191 24780 raft_consensus.cc:2272] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.125806 24780 ts_tablet_manager.cc:1916] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:30.131693 23333 leader_election.cc:343] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:30.131819 23333 leader_election.cc:304] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.132473 24822 raft_consensus.cc:2749] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:30.133038 23333 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.133162 23333 leader_election.cc:304] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.133600 24670 raft_consensus.cc:2749] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.135332 23787 tablet_bootstrap.cc:492] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.135536 23787 ts_tablet_manager.cc:1403] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20251023 20:15:30.136365 23787 raft_consensus.cc:359] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.136590 23787 raft_consensus.cc:385] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.136684 23787 raft_consensus.cc:740] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.136993 23787 consensus_queue.cc:260] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.137643 23787 ts_tablet_manager.cc:1434] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:30.137967 23787 tablet_bootstrap.cc:492] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.139396 23787 tablet_bootstrap.cc:654] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.139707 24780 ts_tablet_manager.cc:1929] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.139869 24780 log.cc:1199] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/7540354f8577482785e6341fa9022315
I20251023 20:15:30.140861 24780 ts_tablet_manager.cc:1950] T 7540354f8577482785e6341fa9022315 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.141393 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 115) took 1559 ms. Trace:
I20251023 20:15:30.141515 24780 rpcz_store.cc:276] 1023 20:15:28.582228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.582444 (+ 216us) service_pool.cc:225] Handling call
1023 20:15:30.124612 (+1542168us) ts_tablet_manager.cc:1151] Deleting tablet 7540354f8577482785e6341fa9022315
1023 20:15:30.124623 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.129124 (+ 4501us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.139597 (+ 10473us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.141378 (+ 1781us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542028}
I20251023 20:15:30.142005 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 7540354f8577482785e6341fa9022315 (table default.Table_410a6baf06a642b18af37bebddac662f [id=75026c75215e419983355275cf38f7f3]) successfully deleted
I20251023 20:15:30.142100 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 116) took 1473 ms. Trace:
I20251023 20:15:30.142273 24780 rpcz_store.cc:276] 1023 20:15:28.668208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.669187 (+ 979us) service_pool.cc:225] Handling call
1023 20:15:30.141971 (+1472784us) ts_tablet_manager.cc:1151] Deleting tablet 39a62c97e0a3459e91c24c1c2f92389a
1023 20:15:30.141980 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.142085 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472554}
I20251023 20:15:30.142756 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 117) took 1473 ms. Trace:
W20251023 20:15:30.142745 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
I20251023 20:15:30.142891 24780 rpcz_store.cc:276] 1023 20:15:28.669564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.669868 (+ 304us) service_pool.cc:225] Handling call
1023 20:15:30.142664 (+1472796us) ts_tablet_manager.cc:1151] Deleting tablet a841ffdc6cdd432dac7afe2e2b53f224
1023 20:15:30.142672 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.142744 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472619}
I20251023 20:15:30.143316 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 118) took 1473 ms. Trace:
I20251023 20:15:30.143424 24780 rpcz_store.cc:276] 1023 20:15:28.670235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.671027 (+ 792us) service_pool.cc:225] Handling call
1023 20:15:30.143231 (+1472204us) ts_tablet_manager.cc:1151] Deleting tablet 40c659c273c449939d0dd708d39b393f
1023 20:15:30.143239 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.143303 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472009}
W20251023 20:15:30.143630 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
I20251023 20:15:30.143853 24780 tablet_replica.cc:333] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
W20251023 20:15:30.143905 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
I20251023 20:15:30.144093 24780 raft_consensus.cc:2243] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.144260 24780 raft_consensus.cc:2272] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.144778 24780 ts_tablet_manager.cc:1916] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:30.155707 23471 leader_election.cc:343] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: a636a4cdeb404e9f806d8e166ea57938
I20251023 20:15:30.161116 24821 raft_consensus.cc:493] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.161298 24821 raft_consensus.cc:515] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.161907 24821 leader_election.cc:290] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:30.162911 23333 leader_election.cc:343] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 25079532547d4175802b84877790662d
I20251023 20:15:30.163686 24658 raft_consensus.cc:493] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.163817 24658 raft_consensus.cc:515] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.164355 24658 leader_election.cc:290] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.165306 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f73477a06542c0b215ae86716cdc4e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:30.165944 23605 leader_election.cc:343] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.167258 24658 raft_consensus.cc:493] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.167382 24658 raft_consensus.cc:515] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.167913 24658 leader_election.cc:290] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.169175 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbacaf2fea24405caaf84d59b625fca3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:30.169708 23605 leader_election.cc:343] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.177214 24821 raft_consensus.cc:493] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.177354 24821 raft_consensus.cc:515] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.177933 24821 leader_election.cc:290] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.178618 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5cb6b4e2e5e4a2f9183111cfd54ec90" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.178807 23664 raft_consensus.cc:2468] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.179229 23333 leader_election.cc:304] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.179488 24821 raft_consensus.cc:2804] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.179554 24821 raft_consensus.cc:493] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.179606 24821 raft_consensus.cc:3060] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.164816 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f73477a06542c0b215ae86716cdc4e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.182269 23471 leader_election.cc:304] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:30.168618 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbacaf2fea24405caaf84d59b625fca3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.182571 24996 raft_consensus.cc:2749] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.182766 23471 leader_election.cc:304] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:30.183125 24821 raft_consensus.cc:515] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.183267 24996 raft_consensus.cc:3060] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:30.183616 23604 leader_election.cc:343] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.183717 23604 leader_election.cc:304] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:30.184068 23604 leader_election.cc:343] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.184142 23604 leader_election.cc:304] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.184409 24659 raft_consensus.cc:2749] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.184522 24659 raft_consensus.cc:2749] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.185750 24996 raft_consensus.cc:2749] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.185869 24825 raft_consensus.cc:493] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea6853efbbcc484eb7529c7e9b37a47a)
I20251023 20:15:30.186000 24825 raft_consensus.cc:515] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.186223 24297 raft_consensus.cc:493] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:30.186363 24297 raft_consensus.cc:515] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:30.186542 24825 leader_election.cc:290] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.186686 24825 consensus_queue.cc:1048] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.186843 24297 leader_election.cc:290] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.187503 24821 leader_election.cc:290] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:30.188140 23333 leader_election.cc:343] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 40e0301fb5974ec5b6e8291230a736c2
W20251023 20:15:30.188349 23333 leader_election.cc:343] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ae388da9e4f94ba5af82a67088ab7097
I20251023 20:15:30.188395 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5cb6b4e2e5e4a2f9183111cfd54ec90" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.188581 23664 raft_consensus.cc:3060] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.191000 23664 raft_consensus.cc:2468] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.191339 23333 leader_election.cc:304] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.191614 24821 raft_consensus.cc:2804] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.191802 24821 raft_consensus.cc:697] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.192040 24821 consensus_queue.cc:237] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.196718 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5cb6b4e2e5e4a2f9183111cfd54ec90" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.197129 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25079532547d4175802b84877790662d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.197849 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5cb6b4e2e5e4a2f9183111cfd54ec90" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.211169 23333 consensus_peers.cc:597] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2ae19cc80194363b8a9c96fbafb4ebe. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.213546 24670 consensus_queue.cc:1048] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:30.214392 23333 leader_election.cc:343] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:30.217716 23333 leader_election.cc:343] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 40e0301fb5974ec5b6e8291230a736c2
I20251023 20:15:30.217832 23333 leader_election.cc:304] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:30.218393 23333 leader_election.cc:343] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.218487 23333 leader_election.cc:304] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.218878 24297 raft_consensus.cc:2749] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:30.219089 24297 raft_consensus.cc:2749] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:30.219944 23333 leader_election.cc:343] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: ae388da9e4f94ba5af82a67088ab7097
I20251023 20:15:30.220043 23333 leader_election.cc:304] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.220338 24821 raft_consensus.cc:2749] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:30.223114 24821 consensus_queue.cc:1048] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.231880 24659 raft_consensus.cc:493] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.232160 24659 raft_consensus.cc:515] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.232766 24659 leader_election.cc:290] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.233453 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.234566 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.235124 23604 leader_election.cc:343] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:30.235541 23605 leader_election.cc:343] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.235637 23605 leader_election.cc:304] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.236114 24659 raft_consensus.cc:2749] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.241863 23786 tablet_bootstrap.cc:492] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.242017 23786 ts_tablet_manager.cc:1403] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.136s user 0.015s sys 0.008s
I20251023 20:15:30.242858 23786 raft_consensus.cc:359] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.243031 23786 raft_consensus.cc:385] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.243083 23786 raft_consensus.cc:740] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.245891 23786 consensus_queue.cc:260] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.248325 25019 tablet.cc:1722] T f2a27706ecdb4ab78bb96c10a9f6757f P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:30.248695 23786 ts_tablet_manager.cc:1434] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:30.248976 23786 tablet_bootstrap.cc:492] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.249758 25020 tablet.cc:1722] T f2a27706ecdb4ab78bb96c10a9f6757f P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:30.250378 23786 tablet_bootstrap.cc:654] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.251636 23790 tablet_bootstrap.cc:492] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.251760 23790 ts_tablet_manager.cc:1403] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.131s user 0.000s sys 0.022s
I20251023 20:15:30.253808 23790 raft_consensus.cc:359] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.253975 23790 raft_consensus.cc:385] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.254025 23790 raft_consensus.cc:740] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.254207 23790 consensus_queue.cc:260] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.254665 23790 ts_tablet_manager.cc:1434] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251023 20:15:30.254953 23790 tablet_bootstrap.cc:492] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.256253 23790 tablet_bootstrap.cc:654] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.268234 24659 raft_consensus.cc:493] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.268402 24659 raft_consensus.cc:515] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.269011 24659 leader_election.cc:290] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.269155 24659 raft_consensus.cc:493] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.269248 24659 raft_consensus.cc:515] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.269702 24659 leader_election.cc:290] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.271143 23786 tablet_bootstrap.cc:492] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.271288 23786 ts_tablet_manager.cc:1403] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20251023 20:15:30.271704 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.271967 23786 raft_consensus.cc:359] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.272128 23786 raft_consensus.cc:385] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.272182 23786 raft_consensus.cc:740] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:30.272277 23605 leader_election.cc:343] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.272379 23786 consensus_queue.cc:260] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.272939 23786 ts_tablet_manager.cc:1434] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:30.277539 23786 tablet_bootstrap.cc:492] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.277673 23534 raft_consensus.cc:3060] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.278215 23532 raft_consensus.cc:3060] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.281163 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edaed8c82da4e36a312477b15017460" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.282234 23604 leader_election.cc:343] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.286720 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edaed8c82da4e36a312477b15017460" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.288986 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.289891 23605 leader_election.cc:343] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.290043 23605 leader_election.cc:304] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:30.289886 23604 leader_election.cc:343] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.290473 23604 leader_election.cc:304] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.290730 24659 raft_consensus.cc:2749] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.290899 24659 raft_consensus.cc:2749] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.294109 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:30.295887 24297 consensus_queue.cc:1048] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.296461 24297 consensus_queue.cc:1048] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.297281 24996 raft_consensus.cc:493] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.297426 24996 raft_consensus.cc:515] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.297847 23786 tablet_bootstrap.cc:654] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.298190 24996 leader_election.cc:290] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:30.299131 23470 leader_election.cc:343] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: af899c01b4af46d39d27f984cc3d5940
I20251023 20:15:30.300652 23787 tablet_bootstrap.cc:492] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.300884 23787 ts_tablet_manager.cc:1403] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.163s user 0.004s sys 0.002s
I20251023 20:15:30.301687 23787 raft_consensus.cc:359] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.301916 23787 raft_consensus.cc:385] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.302018 23787 raft_consensus.cc:740] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.302448 23787 consensus_queue.cc:260] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.303017 23787 ts_tablet_manager.cc:1434] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:30.306114 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:30.308120 23786 tablet_bootstrap.cc:492] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.308362 23786 ts_tablet_manager.cc:1403] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20251023 20:15:30.309187 23786 raft_consensus.cc:359] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.309562 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:30.309710 23786 raft_consensus.cc:385] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.310062 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:30.310209 23786 raft_consensus.cc:740] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.317283 23786 consensus_queue.cc:260] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.317790 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
W20251023 20:15:30.317672 23471 leader_election.cc:343] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: af899c01b4af46d39d27f984cc3d5940
I20251023 20:15:30.318109 23471 leader_election.cc:304] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:30.318483 24830 raft_consensus.cc:2749] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.318938 24658 raft_consensus.cc:493] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.319181 24658 raft_consensus.cc:515] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.320446 24658 leader_election.cc:290] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:30.321520 23333 leader_election.cc:343] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.322958 23786 ts_tablet_manager.cc:1434] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20251023 20:15:30.323295 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:30.323549 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:30.324008 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:30.324644 24297 raft_consensus.cc:493] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.324795 24297 raft_consensus.cc:515] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.325045 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:30.325486 24297 leader_election.cc:290] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.326396 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.326854 23604 leader_election.cc:343] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:30.327947 23333 leader_election.cc:343] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 58ae6a6e58274b8baf952eaaefde7127
I20251023 20:15:30.328552 24822 raft_consensus.cc:493] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.328670 24822 raft_consensus.cc:515] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.328698 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ae6a6e58274b8baf952eaaefde7127" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.329201 24822 leader_election.cc:290] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.329418 24822 raft_consensus.cc:493] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea6853efbbcc484eb7529c7e9b37a47a)
I20251023 20:15:30.329442 23786 tablet_bootstrap.cc:492] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.329792 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.330034 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0150ab3778c34c77a7e772dd6e82b32f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.330212 23670 raft_consensus.cc:2468] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:30.330271 23605 leader_election.cc:343] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.330367 23605 leader_election.cc:304] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.330544 23333 leader_election.cc:304] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.330852 23786 tablet_bootstrap.cc:654] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.330950 24297 raft_consensus.cc:2804] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.331027 24297 raft_consensus.cc:493] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.331094 24297 raft_consensus.cc:3060] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.331494 24658 raft_consensus.cc:2749] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.333376 23787 tablet_bootstrap.cc:492] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.333622 24297 raft_consensus.cc:515] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:30.334074 23333 leader_election.cc:343] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.334189 23333 leader_election.cc:304] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.334347 24297 leader_election.cc:290] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.334802 24297 raft_consensus.cc:2749] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.334853 23787 tablet_bootstrap.cc:654] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.334872 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0150ab3778c34c77a7e772dd6e82b32f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.335029 23532 raft_consensus.cc:2468] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.329542 24822 raft_consensus.cc:515] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:30.335491 24822 leader_election.cc:290] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:30.335718 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:30.336620 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:30.336866 25020 tablet.cc:1722] T 3a7306245d7b41cfab795450761144f5 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:30.339145 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:30.342352 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:30.342964 23212 catalog_manager.cc:5649] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:30.343951 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:30.344559 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:30.344987 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:30.345643 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:30.346316 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:30.346906 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:30.347335 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:30.347802 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:30.348302 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:30.348834 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:30.349845 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:30.350957 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:30.356472 25027 tablet.cc:1722] T f2a27706ecdb4ab78bb96c10a9f6757f P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:30.361053 23790 tablet_bootstrap.cc:492] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.361238 23790 ts_tablet_manager.cc:1403] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.106s user 0.006s sys 0.010s
I20251023 20:15:30.362121 23790 raft_consensus.cc:359] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.362289 23790 raft_consensus.cc:385] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.362352 23790 raft_consensus.cc:740] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.362532 23790 consensus_queue.cc:260] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.363041 23790 ts_tablet_manager.cc:1434] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:30.363310 23790 tablet_bootstrap.cc:492] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.364882 23790 tablet_bootstrap.cc:654] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.365363 25027 tablet.cc:1722] T 3a7306245d7b41cfab795450761144f5 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:30.366159 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:30.373171 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:30.373587 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_9c3cb66b734f447e82a09d9e653ea072 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 1 got 0
I20251023 20:15:30.374022 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:30.375138 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:30.375344 23446 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:30.375516 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:30.375691 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:30.376399 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:30.378602 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:30.378816 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:30.379393 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:30.379719 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:30.380079 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:30.380143 23786 tablet_bootstrap.cc:492] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.380270 23786 ts_tablet_manager.cc:1403] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.008s
I20251023 20:15:30.380755 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:30.380985 23786 raft_consensus.cc:359] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.381137 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:30.381183 23786 raft_consensus.cc:385] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.381237 23786 raft_consensus.cc:740] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.381431 23786 consensus_queue.cc:260] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.381949 23786 ts_tablet_manager.cc:1434] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:30.382267 24780 ts_tablet_manager.cc:1929] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.382418 24780 log.cc:1199] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/8289ce37fa074891beb584bf2fff9747
I20251023 20:15:30.382834 24780 ts_tablet_manager.cc:1950] T 8289ce37fa074891beb584bf2fff9747 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.383147 23786 tablet_bootstrap.cc:492] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.385491 23786 tablet_bootstrap.cc:654] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.385604 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:30.386432 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 119) took 1714 ms. Trace:
I20251023 20:15:30.387005 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:30.387099 23787 tablet_bootstrap.cc:492] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.387225 23787 ts_tablet_manager.cc:1403] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.003s
I20251023 20:15:30.387435 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:30.387218 24780 rpcz_store.cc:276] 1023 20:15:28.671689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.671728 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:30.143816 (+1472088us) ts_tablet_manager.cc:1151] Deleting tablet 8289ce37fa074891beb584bf2fff9747
1023 20:15:30.143824 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.304260 (+160436us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.382161 (+ 77901us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.386408 (+ 4247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471944}
I20251023 20:15:30.387945 23787 raft_consensus.cc:359] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.388123 23787 raft_consensus.cc:385] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.388172 23787 raft_consensus.cc:740] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.388444 23787 consensus_queue.cc:260] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.388703 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 120) took 1716 ms. Trace:
I20251023 20:15:30.388831 24780 rpcz_store.cc:276] 1023 20:15:28.672027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.672064 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:30.387988 (+1715924us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:30.387998 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.388099 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1715759}
I20251023 20:15:30.388993 23787 ts_tablet_manager.cc:1434] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:30.389288 23787 tablet_bootstrap.cc:492] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.389853 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 121) took 1717 ms. Trace:
I20251023 20:15:30.390617 23787 tablet_bootstrap.cc:654] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.391094 23218 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad] alter complete (version 1)
W20251023 20:15:30.389546 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 55478.9us
I20251023 20:15:30.392572 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 8289ce37fa074891beb584bf2fff9747 (table default.Table_0197c34b4744436b8ba95be1ba33be9f [id=b0b39a6a41394aaab2423e84baa0a063]) successfully deleted
W20251023 20:15:30.392815 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
I20251023 20:15:30.393512 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0150ab3778c34c77a7e772dd6e82b32f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.393682 23670 raft_consensus.cc:3060] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.394354 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0150ab3778c34c77a7e772dd6e82b32f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.394510 23533 raft_consensus.cc:3060] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.394572 25019 tablet.cc:1722] T 3a7306245d7b41cfab795450761144f5 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:30.396157 23670 raft_consensus.cc:2468] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:30.396889 23333 leader_election.cc:343] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: e418a62d00824cff89b37a7bace5c469
W20251023 20:15:30.397250 23333 leader_election.cc:343] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e418a62d00824cff89b37a7bace5c469
I20251023 20:15:30.397574 23333 leader_election.cc:304] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.397610 24830 raft_consensus.cc:493] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.397732 24830 raft_consensus.cc:515] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.398046 24821 raft_consensus.cc:2749] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:30.398284 24830 leader_election.cc:290] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.398679 24780 rpcz_store.cc:276] 1023 20:15:28.672379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.672416 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:30.389729 (+1717313us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:30.389737 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.389836 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1717144}
I20251023 20:15:30.398984 24780 tablet_replica.cc:333] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:30.399158 24780 raft_consensus.cc:2243] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.399291 24780 raft_consensus.cc:2272] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.399808 24780 ts_tablet_manager.cc:1916] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.400848 25040 raft_consensus.cc:493] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.400965 25040 raft_consensus.cc:515] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.401598 25040 leader_election.cc:290] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:30.401924 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
I20251023 20:15:30.402431 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8eaa2167f82475ca2f6a110aba288ec" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.402628 23530 raft_consensus.cc:2468] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.402650 23333 leader_election.cc:304] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:30.402899 23333 leader_election.cc:343] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a8eaa2167f82475ca2f6a110aba288ec
I20251023 20:15:30.403131 24821 raft_consensus.cc:2804] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.403235 24821 raft_consensus.cc:697] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.403451 24821 consensus_queue.cc:237] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.403601 23533 raft_consensus.cc:2468] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:30.404044 23470 leader_election.cc:343] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 46f700ea3db04a4bad539c943e850a44
I20251023 20:15:30.404861 23333 leader_election.cc:304] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:30.404844 23471 leader_election.cc:343] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 46f700ea3db04a4bad539c943e850a44
I20251023 20:15:30.405506 24825 raft_consensus.cc:2804] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.405628 24825 raft_consensus.cc:493] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.405746 24825 raft_consensus.cc:3060] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.405820 23211 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 56) took 4739 ms. Trace:
I20251023 20:15:30.405265 23471 leader_election.cc:304] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:30.405952 23211 rpcz_store.cc:276] 1023 20:15:25.665957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.508871 (+2842914us) service_pool.cc:225] Handling call
1023 20:15:28.509288 (+ 417us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:28.557512 (+ 48224us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:28.690456 (+132944us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:28.690568 (+ 112us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:28.720376 (+ 29808us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:30.405769 (+1685393us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:28.691205 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.691330 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.691338 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:28.691340 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:28.691448 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.691464 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.691470 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.691542 (+ 72us) tablet.cc:699] Row locks acquired
1023 20:15:28.691545 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:28.691581 (+ 36us) write_op.cc:270] Start()
1023 20:15:28.691616 (+ 35us) write_op.cc:276] Timestamp: P: 1761250528691556 usec, L: 0
1023 20:15:28.691620 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.708575 (+ 16955us) log.cc:844] Serialized 405 byte log entry
1023 20:15:28.708779 (+ 204us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 000000000098330a 00007f899878e682 00007f8999b70ca5 00007f8999b6f7c9 00007f8999b77bf0 00007f8999b779fd 0000000000a880ec 00007f898c95576f 00007f898c958a3b 00007f898c9588b8 0000000000a880ec
1023 20:15:28.719536 (+ 10757us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.719773 (+ 237us) write_op.cc:301] APPLY: starting
1023 20:15:28.719789 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.719831 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.719834 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.719897 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.719928 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.719935 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:28.720025 (+ 90us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.720087 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.720109 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:28.720116 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.730437 (+ 10321us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.730448 (+ 11us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":7216,"prepare.run_wall_time_us":39214,"raft.queue_time_us":2416,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":27878,"spinlock_wait_cycles":38336384,"thread_start_us":21540,"threads_started":1}]]}
I20251023 20:15:30.407326 24996 raft_consensus.cc:2749] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.408245 24825 raft_consensus.cc:515] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.408856 24825 leader_election.cc:290] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.409201 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c] alter complete (version 3)
W20251023 20:15:30.413017 23216 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92c50c4845524163834861438291a3ec consensus state has no leader
I20251023 20:15:30.416623 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.416913 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.417634 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.417833 23216 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 62) took 4386 ms. Trace:
I20251023 20:15:30.417944 23216 rpcz_store.cc:276] 1023 20:15:26.026442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.104786 (+3078344us) service_pool.cc:225] Handling call
1023 20:15:29.105217 (+ 431us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:29.105413 (+ 196us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:29.105618 (+ 205us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:29.105689 (+ 71us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:29.112915 (+ 7226us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:30.413190 (+1300275us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:29.106170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:29.106261 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:29.106267 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:29.106269 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:29.106356 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:29.106368 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:29.106373 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:29.106426 (+ 53us) tablet.cc:699] Row locks acquired
1023 20:15:29.106428 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:29.106457 (+ 29us) write_op.cc:270] Start()
1023 20:15:29.106485 (+ 28us) write_op.cc:276] Timestamp: P: 1761250529106437 usec, L: 0
1023 20:15:29.106488 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:29.106650 (+ 162us) log.cc:844] Serialized 405 byte log entry
1023 20:15:29.111487 (+ 4837us) op_driver.cc:464] REPLICATION: finished
1023 20:15:29.112468 (+ 981us) write_op.cc:301] APPLY: starting
1023 20:15:29.112482 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:29.112516 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:29.112547 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:29.112591 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:29.112620 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:29.112627 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:29.112690 (+ 63us) log.cc:844] Serialized 23 byte log entry
1023 20:15:29.112750 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.112766 (+ 16us) write_op.cc:454] Released schema lock
1023 20:15:29.112797 (+ 31us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:29.112865 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.112870 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":904,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":393,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":590,"prepare.run_wall_time_us":589,"raft.queue_time_us":893,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":251,"replication_time_us":4969}]]}
I20251023 20:15:30.420147 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.420276 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.420328 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.420435 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.420567 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.420692 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.420946 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:30.426255 23215 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4d71f0cfdb6449fbd74b854c15326d6 consensus state has no leader
I20251023 20:15:30.427585 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.427837 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.429925 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8eaa2167f82475ca2f6a110aba288ec" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.430124 23530 raft_consensus.cc:3060] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:30.431491 23333 leader_election.cc:343] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a8eaa2167f82475ca2f6a110aba288ec
I20251023 20:15:30.432704 23530 raft_consensus.cc:2468] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.441656 23215 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 65) took 3780 ms. Trace:
I20251023 20:15:30.441778 23215 rpcz_store.cc:276] 1023 20:15:26.658865 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.198189 (+2539324us) service_pool.cc:225] Handling call
1023 20:15:29.198551 (+ 362us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:29.198776 (+ 225us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:29.198951 (+ 175us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:29.199013 (+ 62us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:29.209120 (+ 10107us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.209126 (+ 6us) write_op.cc:454] Released schema lock
1023 20:15:29.209193 (+ 67us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:30.439040 (+1229847us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:29.199478 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:29.199596 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:29.199601 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:29.199603 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:29.199689 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:29.199700 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:29.199705 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:29.199778 (+ 73us) tablet.cc:699] Row locks acquired
1023 20:15:29.199781 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:29.199813 (+ 32us) write_op.cc:270] Start()
1023 20:15:29.199841 (+ 28us) write_op.cc:276] Timestamp: P: 1761250529199785 usec, L: 0
1023 20:15:29.199844 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:29.200036 (+ 192us) log.cc:844] Serialized 405 byte log entry
1023 20:15:29.203431 (+ 3395us) op_driver.cc:464] REPLICATION: finished
1023 20:15:29.203595 (+ 164us) write_op.cc:301] APPLY: starting
1023 20:15:29.203609 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:29.203649 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:29.203675 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:29.203728 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:29.203760 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:29.203767 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:29.203838 (+ 71us) log.cc:844] Serialized 23 byte log entry
1023 20:15:29.203905 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.203924 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:29.203931 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":12579,"child_traces":[["op",{"apply.queue_time_us":118,"apply.run_cpu_time_us":384,"apply.run_wall_time_us":383,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":646,"raft.queue_time_us":188,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":199,"replication_time_us":3559}]]}
I20251023 20:15:30.445899 23333 leader_election.cc:304] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:30.446847 24670 raft_consensus.cc:2804] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.446975 24670 raft_consensus.cc:697] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.447230 24670 consensus_queue.cc:237] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.448915 24670 consensus_queue.cc:1048] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.450908 24996 raft_consensus.cc:493] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.451040 24996 raft_consensus.cc:515] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.451629 24996 leader_election.cc:290] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.452972 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:30.452840 23333 consensus_peers.cc:597] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 994a555b24e4432a84601c54d422aa5d. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.454232 24670 consensus_queue.cc:1048] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:30.456419 23203 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92c50c4845524163834861438291a3ec consensus state has no leader
I20251023 20:15:30.460202 24670 raft_consensus.cc:493] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.460435 24670 raft_consensus.cc:515] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.461135 24670 leader_election.cc:290] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.461167 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.461378 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.461561 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.461694 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.461719 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.461853 23638 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.461899 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462028 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462049 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.461836 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462175 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462234 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462318 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462433 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462324 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.462963 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:30.473230 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a86a25659ec44d7c825783036342ed45" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.473703 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b680b1cbc103403b91a1104248ecbdcf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:30.476593 23333 leader_election.cc:343] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a86a25659ec44d7c825783036342ed45
I20251023 20:15:30.476934 23471 leader_election.cc:304] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:30.477372 24996 raft_consensus.cc:2749] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.479888 23786 tablet_bootstrap.cc:492] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.480016 23786 ts_tablet_manager.cc:1403] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20251023 20:15:30.480746 23786 raft_consensus.cc:359] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.480921 23786 raft_consensus.cc:385] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.480970 23786 raft_consensus.cc:740] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.481215 23786 consensus_queue.cc:260] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.481681 23786 ts_tablet_manager.cc:1434] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251023 20:15:30.481784 23333 leader_election.cc:343] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.481917 23333 leader_election.cc:304] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.481958 23786 tablet_bootstrap.cc:492] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.482136 24825 raft_consensus.cc:2749] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.483500 23786 tablet_bootstrap.cc:654] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.496315 23790 tablet_bootstrap.cc:492] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.496479 23790 ts_tablet_manager.cc:1403] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.023s
I20251023 20:15:30.497500 23790 raft_consensus.cc:359] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.497706 23790 raft_consensus.cc:385] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.497782 23790 raft_consensus.cc:740] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.498008 23790 consensus_queue.cc:260] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.498509 23790 ts_tablet_manager.cc:1434] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:30.502368 23790 tablet_bootstrap.cc:492] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.504297 23790 tablet_bootstrap.cc:654] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.510728 23787 tablet_bootstrap.cc:492] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.510929 23787 ts_tablet_manager.cc:1403] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.122s user 0.002s sys 0.000s
I20251023 20:15:30.511713 23787 raft_consensus.cc:359] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.511910 23787 raft_consensus.cc:385] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.511962 23787 raft_consensus.cc:740] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.512194 23787 consensus_queue.cc:260] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.514636 23787 ts_tablet_manager.cc:1434] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251023 20:15:30.514946 23787 tablet_bootstrap.cc:492] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.516623 23787 tablet_bootstrap.cc:654] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.516675 24996 raft_consensus.cc:493] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.516817 24996 raft_consensus.cc:515] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:30.517412 24996 leader_election.cc:290] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:30.520871 23471 leader_election.cc:343] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 600f0110f8bd4b05983d5922d871c266
W20251023 20:15:30.521227 23470 leader_election.cc:343] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 600f0110f8bd4b05983d5922d871c266
I20251023 20:15:30.521332 23470 leader_election.cc:304] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:30.521656 24996 raft_consensus.cc:2749] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:30.524557 23333 consensus_peers.cc:597] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d951134f6773410386bc57f7f90310c3. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.525234 23790 tablet_bootstrap.cc:492] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.525417 23790 ts_tablet_manager.cc:1403] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20251023 20:15:30.526208 23790 raft_consensus.cc:359] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.526437 23790 raft_consensus.cc:385] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.526536 23790 raft_consensus.cc:740] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.526837 23790 consensus_queue.cc:260] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.527657 23790 ts_tablet_manager.cc:1434] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:30.528456 23790 tablet_bootstrap.cc:492] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.533334 23790 tablet_bootstrap.cc:654] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.533233 24780 ts_tablet_manager.cc:1929] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.536999 24780 log.cc:1199] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/ae388da9e4f94ba5af82a67088ab7097
I20251023 20:15:30.539856 24780 ts_tablet_manager.cc:1950] T ae388da9e4f94ba5af82a67088ab7097 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.546113 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 122) took 1873 ms. Trace:
I20251023 20:15:30.547123 24780 rpcz_store.cc:276] 1023 20:15:28.672744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.672781 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:30.398938 (+1726157us) ts_tablet_manager.cc:1151] Deleting tablet ae388da9e4f94ba5af82a67088ab7097
1023 20:15:30.398947 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.451378 (+ 52431us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.532196 (+ 80818us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.546082 (+ 13886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1725993}
I20251023 20:15:30.553663 24780 tablet_replica.cc:333] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:30.555218 24780 raft_consensus.cc:2243] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.556550 24780 raft_consensus.cc:2272] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.560226 25045 tablet_replica.cc:333] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:30.560570 25045 raft_consensus.cc:2243] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.563498 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet ae388da9e4f94ba5af82a67088ab7097 (table default.Table_7a4278caa64d4373b42c8b865ea6c2d8 [id=8f99b8c48cce4b63af995ef9ddb1edfa]) successfully deleted
I20251023 20:15:30.572000 24780 ts_tablet_manager.cc:1916] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:30.575868 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
I20251023 20:15:30.576839 24825 raft_consensus.cc:493] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.576982 24825 raft_consensus.cc:515] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.577613 24825 leader_election.cc:290] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.578454 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.578981 23333 leader_election.cc:343] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.579806 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 8e541aeb3cb54d19a673b541a8a9c127 (DEFAULT_TABLE table=default.Table_d55345cd0dd44edaa55e9e412037691f [id=4da4a7df78da46b18a39004985932e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.580221 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e541aeb3cb54d19a673b541a8a9c127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.580430 23369 tablet_service.cc:1505] Processing CreateTablet for tablet 8e541aeb3cb54d19a673b541a8a9c127 (DEFAULT_TABLE table=default.Table_d55345cd0dd44edaa55e9e412037691f [id=4da4a7df78da46b18a39004985932e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.580807 23369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e541aeb3cb54d19a673b541a8a9c127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.583877 23372 tablet_service.cc:1505] Processing CreateTablet for tablet 098af5ca8fa44e939f3b6f7488e09cc3 (DEFAULT_TABLE table=default.Table_c462519a21af4c17bc119f202f50922a [id=7fb99f9e0faa4759ad4c12b5ce1767ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.584226 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098af5ca8fa44e939f3b6f7488e09cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.585333 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 233d4e7864804665a61a9acca305360b (DEFAULT_TABLE table=default.Table_bdcbe2f8128a4a89b305439e5123c2fe [id=452652d5f027441cb6f821e5d1db0c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.585662 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233d4e7864804665a61a9acca305360b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.587018 23501 tablet_service.cc:1505] Processing CreateTablet for tablet 261834d00bd54fbbb3ea0b3cd44bc393 (DEFAULT_TABLE table=default.Table_dce798c03c814b1899812dc8c06f4452 [id=6f9fc2e5ee6146e3a074b5fbd14314f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.587412 23501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 261834d00bd54fbbb3ea0b3cd44bc393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.588943 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 261834d00bd54fbbb3ea0b3cd44bc393 (DEFAULT_TABLE table=default.Table_dce798c03c814b1899812dc8c06f4452 [id=6f9fc2e5ee6146e3a074b5fbd14314f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.589356 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 348fc49768fb4bb4bfc6a4b5beedcdd5 (DEFAULT_TABLE table=default.Table_03569553d00b42c890bd05b75ff28289 [id=a5b078d079dc454ab853c988889beb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.589380 23508 tablet_service.cc:1505] Processing CreateTablet for tablet 348fc49768fb4bb4bfc6a4b5beedcdd5 (DEFAULT_TABLE table=default.Table_03569553d00b42c890bd05b75ff28289 [id=a5b078d079dc454ab853c988889beb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.589668 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348fc49768fb4bb4bfc6a4b5beedcdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.589716 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348fc49768fb4bb4bfc6a4b5beedcdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.589825 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 233d4e7864804665a61a9acca305360b (DEFAULT_TABLE table=default.Table_bdcbe2f8128a4a89b305439e5123c2fe [id=452652d5f027441cb6f821e5d1db0c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.590096 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233d4e7864804665a61a9acca305360b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.593402 23506 tablet_service.cc:1505] Processing CreateTablet for tablet 098af5ca8fa44e939f3b6f7488e09cc3 (DEFAULT_TABLE table=default.Table_c462519a21af4c17bc119f202f50922a [id=7fb99f9e0faa4759ad4c12b5ce1767ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.593726 23506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098af5ca8fa44e939f3b6f7488e09cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.595245 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 261834d00bd54fbbb3ea0b3cd44bc393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.611052 23790 tablet_bootstrap.cc:492] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.611311 23790 ts_tablet_manager.cc:1403] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.083s user 0.024s sys 0.005s
I20251023 20:15:30.611670 24780 ts_tablet_manager.cc:1929] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.611886 24780 log.cc:1199] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/f4778436aed64b53adb2bab54bcf263b
I20251023 20:15:30.612337 23790 raft_consensus.cc:359] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.612669 23790 raft_consensus.cc:385] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.612764 23790 raft_consensus.cc:740] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.612483 24780 ts_tablet_manager.cc:1950] T f4778436aed64b53adb2bab54bcf263b P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.613134 23790 consensus_queue.cc:260] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.613457 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 123) took 1940 ms. Trace:
I20251023 20:15:30.613547 24780 rpcz_store.cc:276] 1023 20:15:28.673126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.673167 (+ 41us) service_pool.cc:225] Handling call
1023 20:15:30.553163 (+1879996us) ts_tablet_manager.cc:1151] Deleting tablet f4778436aed64b53adb2bab54bcf263b
1023 20:15:30.553177 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.591622 (+ 38445us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.611481 (+ 19859us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.613441 (+ 1960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879564}
I20251023 20:15:30.614012 24780 tablet_replica.cc:333] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:30.614243 24780 raft_consensus.cc:2243] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.614387 24780 raft_consensus.cc:2272] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.614921 24780 ts_tablet_manager.cc:1916] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.614886 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet f4778436aed64b53adb2bab54bcf263b (table default.Table_5abd17a4558f4f81a956cdf6df0d4524 [id=46e81c2c4c164452980264259fcff377]) successfully deleted
I20251023 20:15:30.614023 23790 ts_tablet_manager.cc:1434] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:30.617342 23790 tablet_bootstrap.cc:492] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.621646 23790 tablet_bootstrap.cc:654] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.626650 24780 ts_tablet_manager.cc:1929] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.626765 24780 log.cc:1199] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a2dbb7997d0f4e6bac48cfb196f5ccb6
I20251023 20:15:30.627072 24780 ts_tablet_manager.cc:1950] T a2dbb7997d0f4e6bac48cfb196f5ccb6 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:30.627377 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 124) took 1953 ms. Trace:
I20251023 20:15:30.627458 24780 rpcz_store.cc:276] 1023 20:15:28.673769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.673815 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:30.613968 (+1940153us) ts_tablet_manager.cc:1151] Deleting tablet a2dbb7997d0f4e6bac48cfb196f5ccb6
1023 20:15:30.613977 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.623444 (+ 9467us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.626552 (+ 3108us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.627358 (+ 806us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1939962}
I20251023 20:15:30.627938 23218 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_b33239cad93e4f27aee7c9dbe7fab6d9 [id=910ad9776933488eb34806b4aaff92d8] alter complete (version 1)
I20251023 20:15:30.629052 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 94631fca1b834bc89791cb07fdc93523 (DEFAULT_TABLE table=default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.629511 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94631fca1b834bc89791cb07fdc93523. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.631772 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 (table default.Table_e186a061f84042cd82645b37beae1da1 [id=dc553d161a13467c99df49720181dba1]) successfully deleted
I20251023 20:15:30.631667 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 125) took 1954 ms. Trace:
I20251023 20:15:30.633977 24780 rpcz_store.cc:276] 1023 20:15:28.677424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.677466 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:30.631234 (+1953768us) ts_tablet_manager.cc:1151] Deleting tablet c1f9935f5e4f4daa9df8fe1611ed76c9
1023 20:15:30.631412 (+ 178us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.631638 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953588}
W20251023 20:15:30.635046 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
I20251023 20:15:30.568130 24658 raft_consensus.cc:493] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.577280 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.643483 24658 raft_consensus.cc:515] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.643590 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.643754 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.643874 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.643985 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.577414 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.644156 24658 leader_election.cc:290] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:30.644438 23790 tablet_bootstrap.cc:492] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.644625 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.644655 23790 ts_tablet_manager.cc:1403] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20251023 20:15:30.644796 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.644919 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.645035 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.645190 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.645125 25019 tablet.cc:1722] T 0232c33db59641b3af27bfd15cd7fb81 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251023 20:15:30.645313 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.645429 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.645555 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.645694 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.577592 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.577760 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.577908 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.578014 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.578102 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.578187 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.578644 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.578953 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.646059 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72408899f710470baaf6fdb268cb04c8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.646180 23665 raft_consensus.cc:2468] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.646245 23395 raft_consensus.cc:2468] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:30.646128 23790 raft_consensus.cc:359] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.598233 23640 tablet_service.cc:1505] Processing CreateTablet for tablet 8e541aeb3cb54d19a673b541a8a9c127 (DEFAULT_TABLE table=default.Table_d55345cd0dd44edaa55e9e412037691f [id=4da4a7df78da46b18a39004985932e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.646337 23790 raft_consensus.cc:385] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.646441 23790 raft_consensus.cc:740] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.646633 23640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e541aeb3cb54d19a673b541a8a9c127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.646674 23605 leader_election.cc:304] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.644165 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.647011 23787 tablet_bootstrap.cc:492] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.647120 23787 ts_tablet_manager.cc:1403] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.132s user 0.000s sys 0.003s
I20251023 20:15:30.646764 23790 consensus_queue.cc:260] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.647838 23787 raft_consensus.cc:359] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.647971 23787 raft_consensus.cc:385] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.648010 23787 raft_consensus.cc:740] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.648154 23787 consensus_queue.cc:260] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.648473 25045 raft_consensus.cc:2272] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.649060 25045 ts_tablet_manager.cc:1916] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.648972 23790 ts_tablet_manager.cc:1434] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:30.598562 23638 tablet_service.cc:1505] Processing CreateTablet for tablet 233d4e7864804665a61a9acca305360b (DEFAULT_TABLE table=default.Table_bdcbe2f8128a4a89b305439e5123c2fe [id=452652d5f027441cb6f821e5d1db0c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.649446 23638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 233d4e7864804665a61a9acca305360b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.598865 23642 tablet_service.cc:1505] Processing CreateTablet for tablet 261834d00bd54fbbb3ea0b3cd44bc393 (DEFAULT_TABLE table=default.Table_dce798c03c814b1899812dc8c06f4452 [id=6f9fc2e5ee6146e3a074b5fbd14314f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.653481 23642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 261834d00bd54fbbb3ea0b3cd44bc393. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.658174 25040 raft_consensus.cc:493] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.658304 25040 raft_consensus.cc:515] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.658887 25040 leader_election.cc:290] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.599116 23633 tablet_service.cc:1505] Processing CreateTablet for tablet 348fc49768fb4bb4bfc6a4b5beedcdd5 (DEFAULT_TABLE table=default.Table_03569553d00b42c890bd05b75ff28289 [id=a5b078d079dc454ab853c988889beb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.659255 23633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348fc49768fb4bb4bfc6a4b5beedcdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.599366 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 098af5ca8fa44e939f3b6f7488e09cc3 (DEFAULT_TABLE table=default.Table_c462519a21af4c17bc119f202f50922a [id=7fb99f9e0faa4759ad4c12b5ce1767ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.661350 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098af5ca8fa44e939f3b6f7488e09cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.664242 25027 tablet.cc:1722] T 0232c33db59641b3af27bfd15cd7fb81 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251023 20:15:30.619115 23636 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.619774 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.620559 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.621234 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.621338 24659 raft_consensus.cc:493] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.665424 24659 raft_consensus.cc:515] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.665966 24659 leader_election.cc:290] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:30.666095 24659 raft_consensus.cc:2804] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.666143 24659 raft_consensus.cc:493] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.666217 24659 raft_consensus.cc:3060] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.668450 24659 raft_consensus.cc:515] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.669572 23333 leader_election.cc:304] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.670375 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed0aa7e36e64843ad6b2ad4b415832a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.670500 23530 raft_consensus.cc:3060] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.670575 24670 raft_consensus.cc:2804] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.670655 24670 raft_consensus.cc:493] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.670717 24670 raft_consensus.cc:3060] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.673137 24670 raft_consensus.cc:515] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:30.673399 23333 leader_election.cc:343] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.674249 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.673679 24670 leader_election.cc:290] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:30.674708 23333 leader_election.cc:343] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.674583 25040 consensus_queue.cc:1048] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.681211 25040 raft_consensus.cc:493] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.681357 25040 raft_consensus.cc:515] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.681912 25040 leader_election.cc:290] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.684466 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bfbf0acdc84f2f8fe553694ee9897c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.687343 23333 leader_election.cc:343] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.669015 24659 leader_election.cc:290] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:30.622500 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.621915 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.671479 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed0aa7e36e64843ad6b2ad4b415832a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.674436 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.688366 23670 raft_consensus.cc:3060] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:30.688886 23333 leader_election.cc:343] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: Tablet not RUNNING: STOPPED
I20251023 20:15:30.688992 23333 leader_election.cc:304] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.682680 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bfbf0acdc84f2f8fe553694ee9897c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.689188 23668 raft_consensus.cc:2468] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.691103 23670 raft_consensus.cc:2468] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.691682 25040 raft_consensus.cc:2749] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.692047 23333 leader_election.cc:304] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.692422 23333 leader_election.cc:304] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.692729 25040 raft_consensus.cc:2804] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.692811 25040 raft_consensus.cc:493] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.692870 25040 raft_consensus.cc:3060] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.695286 25040 raft_consensus.cc:515] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.695878 25040 leader_election.cc:290] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.696030 25040 raft_consensus.cc:2804] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.699124 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bfbf0acdc84f2f8fe553694ee9897c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.700490 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bfbf0acdc84f2f8fe553694ee9897c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.700735 23670 raft_consensus.cc:3060] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.703471 23670 raft_consensus.cc:2468] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.705567 25040 raft_consensus.cc:697] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.705780 23333 leader_election.cc:304] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:30.706204 23333 leader_election.cc:343] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.706498 24670 raft_consensus.cc:2804] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.706668 24670 raft_consensus.cc:697] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.706949 24670 consensus_queue.cc:237] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.706416 25040 consensus_queue.cc:237] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.717222 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.721283 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 94631fca1b834bc89791cb07fdc93523 (DEFAULT_TABLE table=default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:30.721700 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94631fca1b834bc89791cb07fdc93523. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.723073 23374 tablet_service.cc:1505] Processing CreateTablet for tablet 94631fca1b834bc89791cb07fdc93523 (DEFAULT_TABLE table=default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]), partition=RANGE (key) PARTITION UNBOUNDED
W20251023 20:15:30.723966 23333 consensus_peers.cc:597] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:30.721287 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.726068 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.726779 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.727034 23787 ts_tablet_manager.cc:1434] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.080s user 0.000s sys 0.001s
W20251023 20:15:30.730046 23604 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 86685.2us
I20251023 20:15:30.730683 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72408899f710470baaf6fdb268cb04c8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.730772 23605 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 65814.4us
I20251023 20:15:30.732277 23787 tablet_bootstrap.cc:492] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.733975 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.734783 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72408899f710470baaf6fdb268cb04c8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.735023 23787 tablet_bootstrap.cc:654] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:30.735383 23604 leader_election.cc:343] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:30.735816 23604 leader_election.cc:343] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.750530 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72408899f710470baaf6fdb268cb04c8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:30.750737 23397 raft_consensus.cc:3060] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.752324 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.753695 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.755156 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.755376 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.755842 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.756075 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.756173 23397 raft_consensus.cc:2468] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
W20251023 20:15:30.758332 23214 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet c7a63d37788240f0a0c3dbfdd14edeac consensus state has no leader
W20251023 20:15:30.759233 23214 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet abdad7d76eb8441496d38642382f74bb consensus state has no leader
I20251023 20:15:30.760254 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.760471 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.760898 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.761137 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.761478 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.761767 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.763379 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 126) took 2085 ms. Trace:
I20251023 20:15:30.763477 24780 rpcz_store.cc:276] 1023 20:15:28.678490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.678543 (+ 53us) service_pool.cc:225] Handling call
1023 20:15:30.634680 (+1956137us) ts_tablet_manager.cc:1151] Deleting tablet 453eea5cdd2e47f29577c61b40ee59c0
1023 20:15:30.750426 (+115746us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.763360 (+ 12934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1955947}
W20251023 20:15:30.763973 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
I20251023 20:15:30.770483 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.770939 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.771131 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.771284 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.772501 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.772630 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.772729 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.772768 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.773018 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.773125 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.773379 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.773636 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.774122 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.774412 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.774617 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:30.774691 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.774945 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.775233 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.775527 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.775808 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.778400 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe with delete_type TABLET_DATA_DELETED (replaced by tablet 94631fca1b834bc89791cb07fdc93523) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.778582 24297 raft_consensus.cc:493] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.778734 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.778712 24297 raft_consensus.cc:515] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.779284 24297 leader_election.cc:290] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.780632 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.781176 23333 leader_election.cc:343] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.782135 23790 tablet_bootstrap.cc:492] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.782167 23374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94631fca1b834bc89791cb07fdc93523. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:30.736730 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.738252 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.739984 25045 ts_tablet_manager.cc:1929] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:30.783993 25045 log.cc:1199] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/9ed0aa7e36e64843ad6b2ad4b415832a
I20251023 20:15:30.784047 23790 tablet_bootstrap.cc:654] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:30.784051 23604 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 48241.1us
W20251023 20:15:30.784247 23604 leader_election.cc:343] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.784304 25045 ts_tablet_manager.cc:1950] T 9ed0aa7e36e64843ad6b2ad4b415832a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:30.785030 25045 tablet_replica.cc:333] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:30.785236 25045 raft_consensus.cc:2243] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.785390 25045 raft_consensus.cc:2272] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.785401 25020 tablet.cc:1722] T 0232c33db59641b3af27bfd15cd7fb81 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251023 20:15:30.785663 24297 raft_consensus.cc:493] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.785794 24297 raft_consensus.cc:515] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.785917 25045 ts_tablet_manager.cc:1916] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.786368 24297 leader_election.cc:290] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.787185 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05933cb9433d48d285b2db82b289bdc8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.787340 23670 raft_consensus.cc:2468] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.787788 23333 leader_election.cc:304] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.788084 24297 raft_consensus.cc:2804] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.788199 24297 raft_consensus.cc:493] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.788275 24297 raft_consensus.cc:3060] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.744830 24658 raft_consensus.cc:493] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.789064 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05933cb9433d48d285b2db82b289bdc8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.789546 23530 raft_consensus.cc:3060] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.790683 24297 raft_consensus.cc:515] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.791244 24297 leader_election.cc:290] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:30.791860 23333 leader_election.cc:343] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.792402 24659 raft_consensus.cc:493] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.792519 24659 raft_consensus.cc:515] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:30.793157 23219 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 190) took 4078 ms (client timeout 1000 ms). Trace:
I20251023 20:15:30.793260 24659 leader_election.cc:290] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.789004 24658 raft_consensus.cc:515] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.793972 24658 leader_election.cc:290] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.794011 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05933cb9433d48d285b2db82b289bdc8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.780109 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.794256 23668 raft_consensus.cc:2468] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.794600 23670 raft_consensus.cc:3060] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.794962 24996 raft_consensus.cc:493] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.795094 24996 raft_consensus.cc:515] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.795594 24996 leader_election.cc:290] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.796658 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac55e14745534a02a2f6a422cab6b149" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.796859 23397 raft_consensus.cc:2468] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:30.797227 23471 leader_election.cc:304] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:30.797737 24996 raft_consensus.cc:2804] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.797817 24996 raft_consensus.cc:493] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.797886 24996 raft_consensus.cc:3060] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.798044 23670 raft_consensus.cc:2468] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.799907 24780 tablet_replica.cc:333] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:30.800184 24780 raft_consensus.cc:2243] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:30.800638 24780 raft_consensus.cc:2272] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.800843 24658 raft_consensus.cc:493] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.800945 24658 raft_consensus.cc:515] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.801543 24658 leader_election.cc:290] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.802219 24780 ts_tablet_manager.cc:1916] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.802454 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:30.802827 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 9ed0aa7e36e64843ad6b2ad4b415832a (table default.Table_42ca9b5fa333476f9d6a35fcb9d0249a [id=70a0aa0e964e425e8ee12c7205b494da]) successfully deleted
W20251023 20:15:30.804046 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
W20251023 20:15:30.804409 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because the tablet was not found. No further retry: Not found: Tablet not found: cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:30.806388 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet e5368813164949ff80e27bf59dcc931a with delete_type TABLET_DATA_DELETED (replaced by tablet abdad7d76eb8441496d38642382f74bb) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.806696 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.806866 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.807025 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.807015 24297 raft_consensus.cc:493] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.807181 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e3d422a66f432789c0cf4a00e57af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.807327 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.807307 24297 raft_consensus.cc:515] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.807461 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.807597 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.807734 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.807875 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808017 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808142 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808274 23643 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808288 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe with delete_type TABLET_DATA_DELETED (replaced by tablet 94631fca1b834bc89791cb07fdc93523) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808398 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808465 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808511 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808696 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.808861 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809011 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809026 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809201 23638 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809357 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809525 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809648 24297 leader_election.cc:290] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.809710 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809849 23636 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.809990 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.810117 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.810237 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.810367 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.810633 24996 raft_consensus.cc:515] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.812191 23333 leader_election.cc:304] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.812466 24297 raft_consensus.cc:2804] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.812619 24297 raft_consensus.cc:493] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.812769 24297 raft_consensus.cc:3060] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.813120 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbacaf2fea24405caaf84d59b625fca3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.813257 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.812641 23333 leader_election.cc:304] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.813794 24821 raft_consensus.cc:2804] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
W20251023 20:15:30.813951 23333 leader_election.cc:343] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.814342 24821 raft_consensus.cc:697] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.814440 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.814628 24821 consensus_queue.cc:237] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:30.814807 23605 leader_election.cc:343] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.814910 23605 leader_election.cc:304] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.815313 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.816027 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:30.816664 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "514cb3b1ed024041ac160a7d9acff618" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.816978 25060 raft_consensus.cc:2749] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.817003 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385900c60d584c15b702f186ecfbc036" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.817895 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a630b83f614447ebf2a08851d6b827d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:30.818073 23605 leader_election.cc:304] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.821475 25060 raft_consensus.cc:2804] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.821759 25060 raft_consensus.cc:697] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:30.822080 25060 consensus_queue.cc:237] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.822367 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbacaf2fea24405caaf84d59b625fca3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.822532 23670 raft_consensus.cc:2468] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:30.824303 23605 leader_election.cc:343] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:30.824574 23605 leader_election.cc:343] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:30.824779 23605 leader_election.cc:343] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.825588 23333 leader_election.cc:304] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.826089 24670 raft_consensus.cc:2804] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.826207 24670 raft_consensus.cc:493] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.826292 24670 raft_consensus.cc:3060] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.828703 24670 raft_consensus.cc:515] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.831487 25045 ts_tablet_manager.cc:1929] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:30.831601 25045 log.cc:1199] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/c9bfbf0acdc84f2f8fe553694ee9897c
I20251023 20:15:30.831885 25045 ts_tablet_manager.cc:1950] T c9bfbf0acdc84f2f8fe553694ee9897c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:30.832396 24670 leader_election.cc:290] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.833351 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbacaf2fea24405caaf84d59b625fca3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.833544 23670 raft_consensus.cc:3060] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.836019 23670 raft_consensus.cc:2468] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.836465 23333 leader_election.cc:304] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.836805 24670 raft_consensus.cc:2804] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.836983 24670 raft_consensus.cc:697] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.837333 24670 consensus_queue.cc:237] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:30.793283 23219 rpcz_store.cc:269] 1023 20:15:26.701894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:27.354313 (+652419us) service_pool.cc:225] Handling call
1023 20:15:28.670350 (+1316037us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.670358 (+ 8us) write_op.cc:454] Released schema lock
1023 20:15:30.780444 (+2110086us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:28.118836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.118975 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.118982 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.118985 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:28.119097 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.119110 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.119115 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.119176 (+ 61us) tablet.cc:699] Row locks acquired
1023 20:15:28.119178 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.119212 (+ 34us) write_op.cc:270] Start()
1023 20:15:28.119247 (+ 35us) write_op.cc:276] Timestamp: P: 1761250528119190 usec, L: 0
1023 20:15:28.119251 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.119429 (+ 178us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.122842 (+ 3413us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.122954 (+ 112us) write_op.cc:301] APPLY: starting
1023 20:15:28.122969 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.123009 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.123042 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.123095 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.123125 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.123131 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:28.123204 (+ 73us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.123261 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.123278 (+ 17us) write_op.cc:454] Released schema lock
1023 20:15:28.123285 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.123356 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.123360 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:28.314789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.314911 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.314918 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.314921 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:28.315018 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.315031 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.315036 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.315098 (+ 62us) tablet.cc:699] Row locks acquired
1023 20:15:28.315100 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.315134 (+ 34us) write_op.cc:270] Start()
1023 20:15:28.315169 (+ 35us) write_op.cc:276] Timestamp: P: 1761250528315099 usec, L: 0
1023 20:15:28.315173 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.315354 (+ 181us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.320069 (+ 4715us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.320183 (+ 114us) write_op.cc:301] APPLY: starting
1023 20:15:28.320198 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.320238 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.320270 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.320321 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.320351 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.320359 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:28.320434 (+ 75us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.320494 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.320512 (+ 18us) write_op.cc:454] Released schema lock
1023 20:15:28.320519 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.320590 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.320594 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:28.407378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.407505 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.407512 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.407515 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:28.407621 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.407636 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.407641 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.407707 (+ 66us) tablet.cc:699] Row locks acquired
1023 20:15:28.407709 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.407742 (+ 33us) write_op.cc:270] Start()
1023 20:15:28.407773 (+ 31us) write_op.cc:276] Timestamp: P: 1761250528407700 usec, L: 0
1023 20:15:28.407785 (+ 12us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.407982 (+ 197us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.422419 (+ 14437us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.422529 (+ 110us) write_op.cc:301] APPLY: starting
1023 20:15:28.422547 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.422590 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.422622 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.422676 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.422708 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.422714 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:28.422808 (+ 94us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.422867 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.422887 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:28.422894 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.422962 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.422966 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:28.455976 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.456130 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.456137 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.456141 (+ 4us) tablet.cc:660] Decoding operations
1023 20:15:28.456252 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.456268 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.456273 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.456341 (+ 68us) tablet.cc:699] Row locks acquired
1023 20:15:28.456344 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:28.456382 (+ 38us) write_op.cc:270] Start()
1023 20:15:28.456422 (+ 40us) write_op.cc:276] Timestamp: P: 1761250528456335 usec, L: 0
1023 20:15:28.456426 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.456624 (+ 198us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.460422 (+ 3798us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.460525 (+ 103us) write_op.cc:301] APPLY: starting
1023 20:15:28.460541 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.460582 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.460608 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.460657 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.460687 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.460694 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:28.460766 (+ 72us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.460830 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.460849 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:28.460856 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:28.460916 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.460921 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:28.652761 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.652862 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.652870 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:28.652873 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:28.652985 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.653001 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.653007 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.653110 (+ 103us) tablet.cc:699] Row locks acquired
1023 20:15:28.653112 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.653153 (+ 41us) write_op.cc:270] Start()
1023 20:15:28.653192 (+ 39us) write_op.cc:276] Timestamp: P: 1761250528653129 usec, L: 0
1023 20:15:28.653195 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.653395 (+ 200us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.657596 (+ 4201us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.658017 (+ 421us) write_op.cc:301] APPLY: starting
1023 20:15:28.658034 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.658131 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.658167 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.658228 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.658262 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.658270 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:28.658387 (+ 117us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.658675 (+ 288us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.658697 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:28.658705 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":222331,"reported_tablets":231,"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":393,"num_ops":1,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":714,"prepare.run_wall_time_us":714,"raft.queue_time_us":894,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":231,"replication_time_us":3562}],["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":396,"apply.run_wall_time_us":396,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":673,"prepare.run_wall_time_us":673,"raft.queue_time_us":1148,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":226,"replication_time_us":4868}],["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":427,"num_ops":1,"prepare.queue_time_us":71,"prepare.run_cpu_time_us":673,"prepare.run_wall_time_us":687,"replication_time_us":14602}],["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":385,"apply.run_wall_time_us":385,"num_ops":1,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":765,"raft.queue_time_us":465,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":235,"replication_time_us":3968}],["op",{"apply.queue_time_us":354,"apply.run_cpu_time_us":483,"apply.run_wall_time_us":747,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":747,"prepare.run_wall_time_us":746,"raft.queue_time_us":949,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":244,"replication_time_us":4369}]]}
I20251023 20:15:30.803376 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05933cb9433d48d285b2db82b289bdc8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.842595 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:30.842727 23333 leader_election.cc:343] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.842845 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a630b83f614447ebf2a08851d6b827d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.843144 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "514cb3b1ed024041ac160a7d9acff618" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.843389 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385900c60d584c15b702f186ecfbc036" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.844153 23787 tablet_bootstrap.cc:492] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.844156 23786 tablet_bootstrap.cc:492] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:30.844269 23787 ts_tablet_manager.cc:1403] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.004s
I20251023 20:15:30.844280 23786 ts_tablet_manager.cc:1403] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.362s user 0.002s sys 0.001s
I20251023 20:15:30.845046 23786 raft_consensus.cc:359] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.845220 23786 raft_consensus.cc:385] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.845268 23786 raft_consensus.cc:740] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.845432 23786 consensus_queue.cc:260] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.844959 23787 raft_consensus.cc:359] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.845656 23787 raft_consensus.cc:385] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.845718 23787 raft_consensus.cc:740] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.845901 23786 ts_tablet_manager.cc:1434] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:30.845888 23787 consensus_queue.cc:260] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.846334 23787 ts_tablet_manager.cc:1434] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:30.846602 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbacaf2fea24405caaf84d59b625fca3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:30.847172 23333 leader_election.cc:343] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.849191 24996 leader_election.cc:290] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.849539 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac55e14745534a02a2f6a422cab6b149" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:30.849725 23397 raft_consensus.cc:3060] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.852645 23397 raft_consensus.cc:2468] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:30.850759 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.851056 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.853957 23787 tablet_bootstrap.cc:492] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:30.854818 23470 leader_election.cc:343] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ac55e14745534a02a2f6a422cab6b149
I20251023 20:15:30.854857 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.855124 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:30.855165 23604 leader_election.cc:343] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.855257 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.855275 23604 leader_election.cc:304] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.855410 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.855572 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:30.855666 23604 leader_election.cc:343] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.855751 23604 leader_election.cc:304] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.855774 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.856115 23787 tablet_bootstrap.cc:654] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:30.856091 23604 leader_election.cc:343] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.856199 23604 leader_election.cc:304] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.856252 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.856446 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.856500 24658 raft_consensus.cc:2749] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.856577 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.856647 24658 raft_consensus.cc:2749] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.856745 24658 raft_consensus.cc:2749] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.856745 23502 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.856880 23515 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.856902 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857018 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857105 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857175 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857196 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857288 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.856917 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857477 23512 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857627 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857642 23471 leader_election.cc:304] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:30.857788 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.857982 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.858160 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.858323 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 951b2276d3324c2d999c8d7fb97767bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.859026 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.859972 23786 tablet_bootstrap.cc:492] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:30.861151 24830 raft_consensus.cc:493] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.861269 24830 raft_consensus.cc:515] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.861560 23786 tablet_bootstrap.cc:654] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.861835 24830 leader_election.cc:290] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.861943 24830 raft_consensus.cc:493] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.862048 24830 raft_consensus.cc:515] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.862449 25045 tablet_replica.cc:333] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:30.862521 24830 leader_election.cc:290] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.862654 24830 raft_consensus.cc:2804] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.862650 25045 raft_consensus.cc:2243] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.862737 24830 raft_consensus.cc:697] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:30.862782 25045 raft_consensus.cc:2272] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.862931 24830 consensus_queue.cc:237] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.863272 25045 ts_tablet_manager.cc:1916] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.828751 24297 raft_consensus.cc:515] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.865595 24297 leader_election.cc:290] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.866132 25068 tablet_replica.cc:333] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:30.866079 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.866238 23670 raft_consensus.cc:3060] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.866318 25068 raft_consensus.cc:2243] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.866447 25068 raft_consensus.cc:2272] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.866961 25068 ts_tablet_manager.cc:1916] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.868193 24780 ts_tablet_manager.cc:1929] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:30.868337 24780 log.cc:1199] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/edec01a63a3046d9ae5a372fc9068b52
I20251023 20:15:30.868703 24780 ts_tablet_manager.cc:1950] T edec01a63a3046d9ae5a372fc9068b52 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:30.872893 23470 leader_election.cc:343] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: ac55e14745534a02a2f6a422cab6b149
W20251023 20:15:30.874439 23217 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 192) took 2098 ms (client timeout 1000 ms). Trace:
W20251023 20:15:30.874707 23470 leader_election.cc:343] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 83dae34129794f49b43038d3b1636ea0
W20251023 20:15:30.874886 23470 leader_election.cc:343] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 352812b56f8d4feb818a364d6adfd86d
I20251023 20:15:30.884052 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
W20251023 20:15:30.875864 23217 rpcz_store.cc:269] 1023 20:15:28.775745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.826393 (+ 50648us) service_pool.cc:225] Handling call
1023 20:15:30.874385 (+2047992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:30.081267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:30.081380 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:30.081387 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:30.081390 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:30.081815 (+ 425us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:30.081829 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:30.081834 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:30.082143 (+ 309us) tablet.cc:699] Row locks acquired
1023 20:15:30.082146 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:30.082196 (+ 50us) write_op.cc:270] Start()
1023 20:15:30.082236 (+ 40us) write_op.cc:276] Timestamp: P: 1761250530082142 usec, L: 0
1023 20:15:30.082240 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:30.082431 (+ 191us) log.cc:844] Serialized 536 byte log entry
1023 20:15:30.162009 (+ 79578us) op_driver.cc:464] REPLICATION: finished
1023 20:15:30.215616 (+ 53607us) write_op.cc:301] APPLY: starting
1023 20:15:30.215634 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:30.215679 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:30.215732 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:30.215788 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:30.215824 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:30.215831 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:30.215911 (+ 80us) log.cc:844] Serialized 23 byte log entry
1023 20:15:30.219556 (+ 3645us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:30.219581 (+ 25us) write_op.cc:454] Released schema lock
1023 20:15:30.219592 (+ 11us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:30.219687 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:30.219690 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":171975,"reported_tablets":225,"child_traces":[["op",{"apply.queue_time_us":53517,"apply.run_cpu_time_us":548,"apply.run_wall_time_us":4071,"num_ops":1,"prepare.queue_time_us":38505,"prepare.run_cpu_time_us":955,"prepare.run_wall_time_us":5106,"raft.queue_time_us":719,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":254,"replication_time_us":79730,"thread_start_us":97177,"threads_started":3}]]}
I20251023 20:15:30.890137 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:30.890693 23333 leader_election.cc:343] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:30.891794 23471 leader_election.cc:343] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 352812b56f8d4feb818a364d6adfd86d
I20251023 20:15:30.891975 23471 leader_election.cc:304] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
W20251023 20:15:30.893055 23471 leader_election.cc:343] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 83dae34129794f49b43038d3b1636ea0
I20251023 20:15:30.893273 23471 leader_election.cc:304] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:30.893635 24996 raft_consensus.cc:2749] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.893798 24996 raft_consensus.cc:2749] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.894435 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet c9bfbf0acdc84f2f8fe553694ee9897c (table default.Table_a0edef92f99c4edebe22ae048c5dcdde [id=e0dec011bafc4166bdd335a6e94475d9]) successfully deleted
I20251023 20:15:30.894968 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:30.895987 25068 ts_tablet_manager.cc:1929] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:30.898475 23670 raft_consensus.cc:2468] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.898959 25068 log.cc:1199] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/48436f3d8f1443db990a2d10b668b70a
I20251023 20:15:30.898949 23333 leader_election.cc:304] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.899506 25068 ts_tablet_manager.cc:1950] T 48436f3d8f1443db990a2d10b668b70a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:30.899742 24670 raft_consensus.cc:2804] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.900002 24670 raft_consensus.cc:697] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.900205 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:30.900345 24670 consensus_queue.cc:237] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.901006 23219 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_cca4260f97404dc2abc827888a312ee4" } modify_external_catalogs: true
I20251023 20:15:30.901311 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:30.901686 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:30.902783 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:30.903287 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:30.904520 25068 tablet_replica.cc:333] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:30.905633 25068 raft_consensus.cc:2243] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:30.906157 25068 raft_consensus.cc:2272] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.906956 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:30.907310 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:30.907593 23219 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_cca4260f97404dc2abc827888a312ee4" } modify_external_catalogs: true
W20251023 20:15:30.907828 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7e08dad1d0f643efb3a821a1b9196290 because the tablet was not found. No further retry: Not found: Tablet not found: 7e08dad1d0f643efb3a821a1b9196290
I20251023 20:15:30.908458 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ae910ac41fd4e349f159ab27e44375a with delete_type TABLET_DATA_DELETED (replaced by tablet 72408899f710470baaf6fdb268cb04c8) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.908655 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.908380 25068 ts_tablet_manager.cc:1916] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.910126 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:30.910307 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:30.911480 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:30.911649 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:30.912707 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:30.913359 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:30.914765 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:30.914953 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:30.916065 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:30.916364 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:30.916623 24670 consensus_queue.cc:1048] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.917271 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:30.917510 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:30.921270 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:30.921306 25045 ts_tablet_manager.cc:1929] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:30.921401 25045 log.cc:1199] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/05933cb9433d48d285b2db82b289bdc8
I20251023 20:15:30.921705 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:30.922573 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:30.922889 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:30.921453 25061 raft_consensus.cc:493] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.923319 25061 raft_consensus.cc:515] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.923477 25045 ts_tablet_manager.cc:1950] T 05933cb9433d48d285b2db82b289bdc8 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:30.925392 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e4189623c7a4ca1bdf73f71663e2b77" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:30.925998 23605 leader_election.cc:343] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.926219 25061 leader_election.cc:290] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:30.927554 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:30.927824 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:30.934149 24670 raft_consensus.cc:493] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.934422 24670 raft_consensus.cc:515] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.935678 24670 leader_election.cc:290] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.936394 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:30.937041 23670 raft_consensus.cc:2468] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:30.937621 23333 leader_election.cc:304] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.937994 24670 raft_consensus.cc:2804] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:30.938124 24670 raft_consensus.cc:493] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:30.938267 24670 raft_consensus.cc:3060] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.941450 24670 raft_consensus.cc:515] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.943298 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.944013 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet 252b5480222f49439052c86a4115cc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.944193 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.944437 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.944468 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:30.945001 25045 tablet_replica.cc:333] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:30.945350 24670 leader_election.cc:290] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.945778 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:30.945984 23670 raft_consensus.cc:3060] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.948441 23670 raft_consensus.cc:2468] T c05b9bcea6ff49188758271401270227 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:30.948928 23333 leader_election.cc:304] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:30.949368 25045 raft_consensus.cc:2243] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:30.949651 25045 pending_rounds.cc:70] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Trying to abort 1 pending ops.
I20251023 20:15:30.949738 25045 pending_rounds.cc:77] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082174250938368 op_type: NO_OP noop_request { }
I20251023 20:15:30.949882 25045 raft_consensus.cc:2889] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:30.950013 25045 raft_consensus.cc:2272] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.950299 24821 raft_consensus.cc:493] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:30.950872 25045 ts_tablet_manager.cc:1916] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:30.951061 24821 raft_consensus.cc:515] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.951913 24670 raft_consensus.cc:2804] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:30.952095 24670 raft_consensus.cc:697] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:30.952353 24670 consensus_queue.cc:237] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:30.953470 23333 leader_election.cc:343] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1e8eb7603c3e46f1b15cdff7a7bbedcf
I20251023 20:15:30.953750 24821 leader_election.cc:290] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.954160 23790 tablet_bootstrap.cc:492] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:30.954303 23790 ts_tablet_manager.cc:1403] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.172s user 0.001s sys 0.010s
I20251023 20:15:30.955122 23790 raft_consensus.cc:359] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.955298 23790 raft_consensus.cc:385] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.955382 23790 raft_consensus.cc:740] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.955622 23790 consensus_queue.cc:260] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.956019 23790 ts_tablet_manager.cc:1434] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:30.956313 23790 tablet_bootstrap.cc:492] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:30.958575 23790 tablet_bootstrap.cc:654] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.944473 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:30.960121 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e4189623c7a4ca1bdf73f71663e2b77" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.960655 23604 leader_election.cc:343] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.960769 23604 leader_election.cc:304] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:30.961169 25061 raft_consensus.cc:2749] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:30.961393 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:30.962478 23333 leader_election.cc:343] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:30.963349 23787 tablet_bootstrap.cc:492] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.963570 23787 ts_tablet_manager.cc:1403] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.006s
I20251023 20:15:30.964356 23787 raft_consensus.cc:359] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.964666 23787 raft_consensus.cc:385] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:30.964751 23787 raft_consensus.cc:740] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:30.964998 23787 consensus_queue.cc:260] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.965654 23787 ts_tablet_manager.cc:1434] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:30.966254 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:30.966789 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:30.967298 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:30.967573 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:30.967828 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:30.968292 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:30.968866 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:30.970826 23532 raft_consensus.cc:3060] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:30.974395 24821 consensus_queue.cc:1048] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:30.980017 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 127) took 2301 ms. Trace:
I20251023 20:15:30.980146 24780 rpcz_store.cc:276] 1023 20:15:28.678908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.679492 (+ 584us) service_pool.cc:225] Handling call
1023 20:15:30.763969 (+2084477us) ts_tablet_manager.cc:1151] Deleting tablet edec01a63a3046d9ae5a372fc9068b52
1023 20:15:30.799856 (+ 35887us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:30.836198 (+ 36342us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.868057 (+ 31859us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:30.979947 (+111890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2083309}
I20251023 20:15:30.980491 24780 tablet_replica.cc:333] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:30.980928 24780 raft_consensus.cc:2243] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:30.981283 24780 pending_rounds.cc:70] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:30.981345 24780 pending_rounds.cc:77] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082155752292352 op_type: NO_OP noop_request { }
I20251023 20:15:30.983299 23787 tablet_bootstrap.cc:492] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:30.983799 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:30.984710 23787 tablet_bootstrap.cc:654] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:30.984855 24780 raft_consensus.cc:2889] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:30.987296 25040 raft_consensus.cc:493] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:30.987459 25040 raft_consensus.cc:515] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:30.988085 25040 leader_election.cc:290] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:30.991439 24780 raft_consensus.cc:2272] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:30.991533 25045 ts_tablet_manager.cc:1929] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251023 20:15:30.942745 23214 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 166) took 2174 ms (client timeout 1000 ms). Trace:
W20251023 20:15:30.942572 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514cb3b1ed024041ac160a7d9acff618 already in progress: creating tablet
W20251023 20:15:30.969029 23213 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 191) took 3163 ms (client timeout 1000 ms). Trace:
I20251023 20:15:30.992204 25045 log.cc:1199] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/72408899f710470baaf6fdb268cb04c8
I20251023 20:15:30.985311 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet edec01a63a3046d9ae5a372fc9068b52 (table default.Table_a4375532322e46288e9b2a62d0a1cdea [id=f543b3d5f2074f96acca26011d0f254d]) successfully deleted
W20251023 20:15:30.993285 23214 rpcz_store.cc:269] 1023 20:15:28.742376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.822600 (+ 80224us) service_pool.cc:225] Handling call
1023 20:15:30.917195 (+2094595us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104398,"reported_tablets":238}
I20251023 20:15:30.993773 23787 tablet_bootstrap.cc:492] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:30.993891 23787 ts_tablet_manager.cc:1403] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20251023 20:15:30.994736 25045 ts_tablet_manager.cc:1950] T 72408899f710470baaf6fdb268cb04c8 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:30.996044 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb01b3f4ff7f41e98063ebf76abeace6" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:30.996678 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:30.996781 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e08dad1d0f643efb3a821a1b9196290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:30.996796 23204 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 76269.3us
I20251023 20:15:30.996970 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:30.997123 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72408899f710470baaf6fdb268cb04c8 already in progress: creating tablet
W20251023 20:15:30.997351 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed0aa7e36e64843ad6b2ad4b415832a already in progress: creating tablet
W20251023 20:15:30.997807 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 95574277dde44983bb0a43d2b5ad01d5 because the tablet was not found. No further retry: Not found: Tablet not found: 95574277dde44983bb0a43d2b5ad01d5
I20251023 20:15:30.997920 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:30.998063 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet cb915d114fc0421999d079c23cad8eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:30.998106 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bfbf0acdc84f2f8fe553694ee9897c already in progress: creating tablet
W20251023 20:15:30.998456 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb01b3f4ff7f41e98063ebf76abeace6 already in progress: creating tablet
W20251023 20:15:30.998662 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7e08dad1d0f643efb3a821a1b9196290 because the tablet was not found. No further retry: Not found: Tablet not found: 7e08dad1d0f643efb3a821a1b9196290
W20251023 20:15:30.998907 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f because the tablet was not found. No further retry: Not found: Tablet not found: ed46b7e67e4e4ae7b30ea69f1d3fdc2f
I20251023 20:15:30.998927 23372 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:30.999116 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05933cb9433d48d285b2db82b289bdc8 already in progress: creating tablet
W20251023 20:15:30.999548 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because the tablet was not found. No further retry: Not found: Tablet not found: cb915d114fc0421999d079c23cad8eb5
I20251023 20:15:30.999745 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 48436f3d8f1443db990a2d10b668b70a (table default.Table_7c03bf4f1b034b139999af6a349dccf0 [id=581d16998437467693ce84962d6717c7]) successfully deleted
W20251023 20:15:31.000597 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet aa794b3bec4b4b3993c814ea7cd02ee2 because the tablet was not found. No further retry: Not found: Tablet not found: aa794b3bec4b4b3993c814ea7cd02ee2
I20251023 20:15:31.000589 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.000723 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:31.000805 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a86a25659ec44d7c825783036342ed45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a86a25659ec44d7c825783036342ed45 already in progress: creating tablet
I20251023 20:15:31.000936 24780 ts_tablet_manager.cc:1916] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:30.996613 23213 rpcz_store.cc:269] 1023 20:15:27.755953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.301870 (+545917us) service_pool.cc:225] Handling call
1023 20:15:29.160743 (+858873us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.160751 (+ 8us) write_op.cc:454] Released schema lock
1023 20:15:30.919779 (+1759028us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:29.139209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:29.139330 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:29.139337 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:29.139340 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:29.139443 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:29.139457 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:29.139462 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:29.139537 (+ 75us) tablet.cc:699] Row locks acquired
1023 20:15:29.139540 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:29.139578 (+ 38us) write_op.cc:270] Start()
1023 20:15:29.139613 (+ 35us) write_op.cc:276] Timestamp: P: 1761250529139555 usec, L: 0
1023 20:15:29.139616 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:29.139838 (+ 222us) log.cc:844] Serialized 536 byte log entry
1023 20:15:29.143387 (+ 3549us) op_driver.cc:464] REPLICATION: finished
1023 20:15:29.143535 (+ 148us) write_op.cc:301] APPLY: starting
1023 20:15:29.143551 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:29.143592 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:29.143616 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:29.143667 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:29.143698 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:29.143705 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:29.143794 (+ 89us) log.cc:844] Serialized 23 byte log entry
1023 20:15:29.143854 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.143872 (+ 18us) write_op.cc:454] Released schema lock
1023 20:15:29.143879 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":79033,"reported_tablets":237,"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":408,"apply.run_wall_time_us":407,"num_ops":1,"prepare.queue_time_us":19249,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":736,"raft.queue_time_us":1136,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":246,"replication_time_us":3738}]]}
I20251023 20:15:31.001492 23213 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_6621b13baee14821a56e8610ed7c68a7" } new_table_name: "default.Table_571807db7b4b4d189e39cd9c352f33c6" modify_external_catalogs: true
I20251023 20:15:31.002221 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.006109 25045 tablet_replica.cc:333] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:31.006506 23333 leader_election.cc:343] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: fb01b3f4ff7f41e98063ebf76abeace6
I20251023 20:15:31.006906 25045 raft_consensus.cc:2243] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.007257 25045 raft_consensus.cc:2272] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.008334 25045 ts_tablet_manager.cc:1916] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.009467 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.013899 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 05933cb9433d48d285b2db82b289bdc8 (table default.Table_f5d04920d5d94f63a2f285d7d824d648 [id=9b498d31bbd945c391ffc62f4948ec85]) successfully deleted
W20251023 20:15:31.014235 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fb01b3f4ff7f41e98063ebf76abeace6 because the tablet was not found. No further retry: Not found: Tablet not found: fb01b3f4ff7f41e98063ebf76abeace6
I20251023 20:15:31.014664 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 72408899f710470baaf6fdb268cb04c8 (table default.Table_4dbe3b1770374c18821af919d529fc09 [id=a4c4486346ac4c9aab7d06a87b919105]) successfully deleted
I20251023 20:15:31.009802 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.017560 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.020215 23786 tablet_bootstrap.cc:492] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.020375 23786 ts_tablet_manager.cc:1403] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.160s user 0.005s sys 0.029s
W20251023 20:15:31.020607 23333 leader_election.cc:343] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.020761 23333 leader_election.cc:304] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.020901 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.021136 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.021200 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet f1a6fe726ef844829a7ffec9d4494637 with delete_type TABLET_DATA_DELETED (replaced by tablet af899c01b4af46d39d27f984cc3d5940) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.021557 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.023126 25061 raft_consensus.cc:493] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.023252 25061 raft_consensus.cc:515] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.023625 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.023813 25061 leader_election.cc:290] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.023862 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b6c3e7169154e3497dd3570dca0d745 with delete_type TABLET_DATA_DELETED (replaced by tablet 19c58a406d564b048fb16a85aa0b754b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.024003 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:31.024746 23333 leader_election.cc:343] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.026998 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.027465 23605 leader_election.cc:343] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.029490 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.029747 23512 tablet_service.cc:1552] Processing DeleteTablet for tablet dafb9db710f94c328186e3179bf0b319 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.029996 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:31.030000 23333 leader_election.cc:343] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 1e8eb7603c3e46f1b15cdff7a7bbedcf
I20251023 20:15:31.030357 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet 3894dd4feeea4c16861a6c9520059474 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.030550 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet e0f8d8184a68453abd2792ccc7740aa1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb915d114fc0421999d079c23cad8eb5) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.031577 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.031657 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 585a9af140d9479fb83d13e68a62300c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.032054 23510 tablet_service.cc:1552] Processing DeleteTablet for tablet 2fa6a838b1bd409ba2a4e2074576ee23 with delete_type TABLET_DATA_DELETED (replaced by tablet aa59141039de402e9ccce427604ecb31) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.032519 24297 raft_consensus.cc:2749] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.032537 23512 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.032680 23507 tablet_service.cc:1552] Processing DeleteTablet for tablet 925e53d9ea6f4c7baa38551eb917828d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:31.032691 23210 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 193) took 3181 ms (client timeout 1000 ms). Trace:
I20251023 20:15:31.032847 23498 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.033003 23513 tablet_service.cc:1552] Processing DeleteTablet for tablet e5368813164949ff80e27bf59dcc931a with delete_type TABLET_DATA_DELETED (replaced by tablet abdad7d76eb8441496d38642382f74bb) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.033169 23333 leader_election.cc:304] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.033244 23501 tablet_service.cc:1552] Processing DeleteTablet for tablet d0e61de7a93d450d9ed79eef82d7e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:31.032797 23210 rpcz_store.cc:269] 1023 20:15:27.851241 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.377353 (+526112us) service_pool.cc:225] Handling call
1023 20:15:31.032648 (+2655295us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:29.515369 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:29.515499 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:29.515506 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:29.515509 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:29.515671 (+ 162us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:29.515685 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:29.515690 (+ 5us) tablet.cc:683] Acquiring locks for 3 operations
1023 20:15:29.515780 (+ 90us) tablet.cc:699] Row locks acquired
1023 20:15:29.515783 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:29.515819 (+ 36us) write_op.cc:270] Start()
1023 20:15:29.515851 (+ 32us) write_op.cc:276] Timestamp: P: 1761250529515769 usec, L: 0
1023 20:15:29.515854 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:29.516071 (+ 217us) log.cc:844] Serialized 1302 byte log entry
1023 20:15:29.519743 (+ 3672us) op_driver.cc:464] REPLICATION: finished
1023 20:15:29.519868 (+ 125us) write_op.cc:301] APPLY: starting
1023 20:15:29.519883 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:29.519968 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:29.519999 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:29.520076 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:29.520109 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1023 20:15:29.520116 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:29.520197 (+ 81us) log.cc:844] Serialized 35 byte log entry
1023 20:15:29.520257 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.520279 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:29.520286 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:29.520352 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:29.520356 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":53409,"reported_tablets":270,"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":476,"apply.run_wall_time_us":476,"num_ops":3,"prepare.queue_time_us":122,"prepare.run_cpu_time_us":809,"prepare.run_wall_time_us":809,"raft.queue_time_us":792,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":228,"replication_time_us":3859}]]}
I20251023 20:15:31.033418 23506 tablet_service.cc:1552] Processing DeleteTablet for tablet 23225d31c2e647398aa58494c627507e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.033581 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.033725 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.034067 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet aa794b3bec4b4b3993c814ea7cd02ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.034250 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.034397 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.034475 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.034550 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:31.034816 23218 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 165) took 3389 ms (client timeout 1000 ms). Trace:
W20251023 20:15:31.034843 23604 leader_election.cc:343] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.034942 23604 leader_election.cc:304] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.034937 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.035317 25060 raft_consensus.cc:2749] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.035319 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.035480 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 86b9f4c5d9e0469c87071a33722ce3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.035614 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.035758 23786 raft_consensus.cc:359] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.035964 23786 raft_consensus.cc:385] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:31.034977 23218 rpcz_store.cc:269] 1023 20:15:27.644897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.007005 (+362108us) service_pool.cc:225] Handling call
1023 20:15:28.855268 (+848263us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.855278 (+ 10us) write_op.cc:454] Released schema lock
1023 20:15:31.034777 (+2179499us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:28.793593 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:28.793714 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:28.793721 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:28.793724 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:28.793842 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:28.793856 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:28.793860 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:28.793926 (+ 66us) tablet.cc:699] Row locks acquired
1023 20:15:28.793928 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:28.793967 (+ 39us) write_op.cc:270] Start()
1023 20:15:28.794003 (+ 36us) write_op.cc:276] Timestamp: P: 1761250528793933 usec, L: 0
1023 20:15:28.794006 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:28.794189 (+ 183us) log.cc:844] Serialized 356 byte log entry
1023 20:15:28.798319 (+ 4130us) op_driver.cc:464] REPLICATION: finished
1023 20:15:28.798401 (+ 82us) write_op.cc:301] APPLY: starting
1023 20:15:28.798418 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:28.798462 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:28.798495 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:28.798561 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:28.798595 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:28.798602 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:28.798681 (+ 79us) log.cc:844] Serialized 23 byte log entry
1023 20:15:28.798740 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:28.798763 (+ 23us) write_op.cc:454] Released schema lock
1023 20:15:28.798778 (+ 15us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1023 20:15:30.469703 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:30.469868 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:30.469876 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:30.469879 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:30.469992 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:30.470007 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:30.470013 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:30.470084 (+ 71us) tablet.cc:699] Row locks acquired
1023 20:15:30.470086 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:30.470128 (+ 42us) write_op.cc:270] Start()
1023 20:15:30.470167 (+ 39us) write_op.cc:276] Timestamp: P: 1761250530470084 usec, L: 0
1023 20:15:30.470171 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:30.470383 (+ 212us) log.cc:844] Serialized 356 byte log entry
1023 20:15:30.473665 (+ 3282us) op_driver.cc:464] REPLICATION: finished
1023 20:15:30.473777 (+ 112us) write_op.cc:301] APPLY: starting
1023 20:15:30.473793 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:30.473842 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:30.473881 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:30.473943 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:30.473978 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:30.473987 (+ 9us) write_op.cc:312] APPLY: finished
1023 20:15:30.474077 (+ 90us) log.cc:844] Serialized 23 byte log entry
1023 20:15:30.474140 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:30.474162 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:30.474169 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:30.474238 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:30.474242 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":183991,"reported_tablets":250,"child_traces":[["op",{"apply.queue_time_us":33,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":431,"num_ops":1,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":708,"raft.queue_time_us":866,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":308,"replication_time_us":4281}],["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":453,"num_ops":1,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":787,"raft.queue_time_us":344,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":283,"replication_time_us":3463}]]}
I20251023 20:15:31.036075 23786 raft_consensus.cc:740] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.036106 24821 raft_consensus.cc:2749] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:31.036309 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet ef1229b9e0384a82811f244d2bf4c8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.036335 23786 consensus_queue.cc:260] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.036454 23373 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.036589 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.036533 23218 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_4ad8e8f4f67e4639bcab0f4fd2c7b099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:31.037393 23218 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ad8e8f4f67e4639bcab0f4fd2c7b099 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:31.037544 23786 ts_tablet_manager.cc:1434] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251023 20:15:31.037861 23786 tablet_bootstrap.cc:492] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.038635 23374 tablet_service.cc:1552] Processing DeleteTablet for tablet 8da82e6d24b14a129b91bb5298139d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.038800 23365 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.039515 23786 tablet_bootstrap.cc:654] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.040256 25068 ts_tablet_manager.cc:1929] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.040376 25068 log.cc:1199] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a8eaa2167f82475ca2f6a110aba288ec
I20251023 20:15:31.040369 23373 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.040673 25068 ts_tablet_manager.cc:1950] T a8eaa2167f82475ca2f6a110aba288ec P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.040886 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet cec51c54ea534b1da4b1d170e79d9f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:31.041030 23212 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 194) took 2004 ms (client timeout 1000 ms). Trace:
I20251023 20:15:31.041108 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 95574277dde44983bb0a43d2b5ad01d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.041218 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.041392 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.041638 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:31.041882 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:31.041890 23372 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.042416 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:31.043195 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:31.043437 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.043746 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_d22911da756748fb8939006df660e885 [id=78fe983663c94eabab93ee902bde6998]). Expected version 5 got 4
I20251023 20:15:31.043798 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet d82c92e863b2470699cc822b8cb8c14d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.044274 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 523d72012c364457b9a2a299f2f789a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.042747 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_9eb13d4c90b44054a9f549bd289a9c6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:31.044894 25068 tablet_replica.cc:333] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:31.044919 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9eb13d4c90b44054a9f549bd289a9c6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:31.045198 25068 raft_consensus.cc:2243] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.045419 25068 raft_consensus.cc:2272] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.046033 25068 ts_tablet_manager.cc:1916] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:31.041180 23212 rpcz_store.cc:269] 1023 20:15:29.036147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.060024 (+ 23877us) service_pool.cc:225] Handling call
1023 20:15:31.040996 (+1980972us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:30.410099 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:30.410236 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:30.410244 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:30.410247 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:30.410368 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:30.410385 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:30.410391 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:30.410464 (+ 73us) tablet.cc:699] Row locks acquired
1023 20:15:30.410467 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:30.410506 (+ 39us) write_op.cc:270] Start()
1023 20:15:30.410544 (+ 38us) write_op.cc:276] Timestamp: P: 1761250530410468 usec, L: 0
1023 20:15:30.410548 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:30.410764 (+ 216us) log.cc:844] Serialized 536 byte log entry
1023 20:15:30.442981 (+ 32217us) op_driver.cc:464] REPLICATION: finished
1023 20:15:30.443200 (+ 219us) write_op.cc:301] APPLY: starting
1023 20:15:30.443214 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:30.443250 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:30.443275 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:30.443320 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:30.443348 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:30.443356 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:30.443423 (+ 67us) log.cc:844] Serialized 23 byte log entry
1023 20:15:30.443474 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:30.443491 (+ 17us) write_op.cc:454] Released schema lock
1023 20:15:30.443498 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:30.443576 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:30.443581 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":57218,"reported_tablets":267,"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":371,"apply.run_wall_time_us":370,"num_ops":1,"prepare.queue_time_us":296,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":787,"raft.queue_time_us":3614,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":7987,"replication_time_us":32398}]]}
I20251023 20:15:31.046262 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a8eaa2167f82475ca2f6a110aba288ec (table default.Table_f4de4bcad2f34502a1b63889f5b5e58a [id=05efdf3ecd3d49d4bc4da65e6f8f5b9b]) successfully deleted
I20251023 20:15:31.046850 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet c97e2e0f839a4ef389fe100b6a81301e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.047065 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.047384 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 1a64a7c2e1f242e1b54243489622bae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.047770 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.047979 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.048302 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a9c5b69df4c04365b1ac1457fe8300cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.046550 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:31.048875 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:31.049324 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e73ae95c73945f081c298f49d47c904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.049485 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.049758 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet f908f086946949829cac15418a584730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.050055 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:31.050093 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.050379 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.049188 23212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_fbff76a175884e39a9efd67edd4db37e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:31.050690 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.050978 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.051257 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 8289ce37fa074891beb584bf2fff9747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:30.994634 23787 raft_consensus.cc:359] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.051491 23787 raft_consensus.cc:385] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.051551 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.051930 25068 ts_tablet_manager.cc:1929] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.052116 25068 log.cc:1199] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/352812b56f8d4feb818a364d6adfd86d
I20251023 20:15:31.052428 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet ae388da9e4f94ba5af82a67088ab7097 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.052510 25068 ts_tablet_manager.cc:1950] T 352812b56f8d4feb818a364d6adfd86d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.053205 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.053385 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:31.053735 23212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbff76a175884e39a9efd67edd4db37e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:31.053874 23787 raft_consensus.cc:740] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.054081 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.054409 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.054790 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d1ddde7e3134e23b3a9356523a93753 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.054705 23787 consensus_queue.cc:260] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.055102 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.051797 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:31.055406 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.055689 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.057870 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:31.059442 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:31.054765 25045 ts_tablet_manager.cc:1929] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.060457 25045 log.cc:1199] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/514cb3b1ed024041ac160a7d9acff618
I20251023 20:15:31.060792 25045 ts_tablet_manager.cc:1950] T 514cb3b1ed024041ac160a7d9acff618 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:31.062115 25045 tablet_replica.cc:333] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:31.062301 25045 raft_consensus.cc:2243] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.062422 25045 raft_consensus.cc:2272] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.062863 25045 ts_tablet_manager.cc:1916] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.064131 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:31.064405 23211 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:31.068545 23787 ts_tablet_manager.cc:1434] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.075s user 0.001s sys 0.001s
I20251023 20:15:31.070518 23787 tablet_bootstrap.cc:492] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.072206 23787 tablet_bootstrap.cc:654] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.073829 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 514cb3b1ed024041ac160a7d9acff618 (table default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]) successfully deleted
W20251023 20:15:31.074136 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a86a25659ec44d7c825783036342ed45 because the tablet was not found. No further retry: Not found: Tablet not found: a86a25659ec44d7c825783036342ed45
W20251023 20:15:31.074349 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 48436f3d8f1443db990a2d10b668b70a because the tablet was not found. No further retry: Not found: Tablet not found: 48436f3d8f1443db990a2d10b668b70a
W20251023 20:15:31.074537 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a636a4cdeb404e9f806d8e166ea57938 because the tablet was not found. No further retry: Not found: Tablet not found: a636a4cdeb404e9f806d8e166ea57938
W20251023 20:15:31.075246 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9f422575c7c4251879130501ab721fd because the tablet was not found. No further retry: Not found: Tablet not found: d9f422575c7c4251879130501ab721fd
W20251023 20:15:31.075467 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 83dae34129794f49b43038d3b1636ea0 because the tablet was not found. No further retry: Not found: Tablet not found: 83dae34129794f49b43038d3b1636ea0
W20251023 20:15:31.075660 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 600f0110f8bd4b05983d5922d871c266 because the tablet was not found. No further retry: Not found: Tablet not found: 600f0110f8bd4b05983d5922d871c266
W20251023 20:15:31.075845 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
W20251023 20:15:31.076047 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c97e2e0f839a4ef389fe100b6a81301e because the tablet was not found. No further retry: Not found: Tablet not found: c97e2e0f839a4ef389fe100b6a81301e
W20251023 20:15:31.076216 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet af899c01b4af46d39d27f984cc3d5940 because the tablet was not found. No further retry: Not found: Tablet not found: af899c01b4af46d39d27f984cc3d5940
W20251023 20:15:31.076440 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
W20251023 20:15:31.076745 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f103bece2fa240c19c23ac51046c13ed because the tablet was not found. No further retry: Not found: Tablet not found: f103bece2fa240c19c23ac51046c13ed
I20251023 20:15:31.079741 25068 tablet_replica.cc:333] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.080041 25068 raft_consensus.cc:2243] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.083895 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 69) took 4376 ms. Trace:
I20251023 20:15:31.084278 24780 ts_tablet_manager.cc:1929] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:31.084445 24780 log.cc:1199] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/6e1b4748c0244b258960b1025da47562
I20251023 20:15:31.084077 23210 rpcz_store.cc:276] 1023 20:15:26.707259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.042579 (+4335320us) service_pool.cc:225] Handling call
1023 20:15:31.044998 (+ 2419us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:31.045475 (+ 477us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:31.083650 (+ 38175us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:31.083756 (+ 106us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:31.083849 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.052831 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.052938 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.052945 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:31.052948 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:31.053041 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.053052 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.053057 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.053156 (+ 99us) tablet.cc:699] Row locks acquired
1023 20:15:31.053159 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:31.053199 (+ 40us) write_op.cc:270] Start()
1023 20:15:31.053236 (+ 37us) write_op.cc:276] Timestamp: P: 1761250531053151 usec, L: 0
1023 20:15:31.053239 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.053424 (+ 185us) log.cc:844] Serialized 433 byte log entry
1023 20:15:31.082964 (+ 29540us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.083177 (+ 213us) write_op.cc:301] APPLY: starting
1023 20:15:31.083190 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.083228 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.083251 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.083387 (+ 136us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.083415 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:31.083422 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:31.083488 (+ 66us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.083539 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.083556 (+ 17us) write_op.cc:454] Released schema lock
1023 20:15:31.083562 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.083629 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.083633 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":444,"num_ops":2,"prepare.queue_time_us":6839,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":725,"replication_time_us":29692}]]}
I20251023 20:15:31.085167 24780 ts_tablet_manager.cc:1950] T 6e1b4748c0244b258960b1025da47562 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:31.085239 25068 raft_consensus.cc:2272] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.085405 23212 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 70) took 4378 ms. Trace:
I20251023 20:15:31.085883 25068 ts_tablet_manager.cc:1916] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.085852 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 129) took 2405 ms. Trace:
I20251023 20:15:31.085566 23212 rpcz_store.cc:276] 1023 20:15:26.707387 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.048311 (+4340924us) service_pool.cc:225] Handling call
1023 20:15:31.054447 (+ 6136us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:31.056130 (+ 1683us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:31.085196 (+ 29066us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:31.085290 (+ 94us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:31.085371 (+ 81us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.073930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.074053 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.074060 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:31.074063 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:31.074149 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.074159 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.074164 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.074237 (+ 73us) tablet.cc:699] Row locks acquired
1023 20:15:31.074240 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:31.074406 (+ 166us) write_op.cc:270] Start()
1023 20:15:31.074465 (+ 59us) write_op.cc:276] Timestamp: P: 1761250531074353 usec, L: 0
1023 20:15:31.074470 (+ 5us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.074672 (+ 202us) log.cc:844] Serialized 433 byte log entry
1023 20:15:31.083031 (+ 8359us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.083688 (+ 657us) write_op.cc:301] APPLY: starting
1023 20:15:31.083697 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.083726 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.083761 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.083832 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.083858 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:31.083864 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:31.083936 (+ 72us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.083975 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.083990 (+ 15us) write_op.cc:454] Released schema lock
1023 20:15:31.083995 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":628,"apply.run_cpu_time_us":361,"apply.run_wall_time_us":361,"num_ops":2,"prepare.queue_time_us":17339,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":822,"replication_time_us":8544,"threads_started":1}]]}
I20251023 20:15:31.085997 24780 rpcz_store.cc:276] 1023 20:15:28.679831 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.679871 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:30.980399 (+2300528us) ts_tablet_manager.cc:1151] Deleting tablet 6e1b4748c0244b258960b1025da47562
1023 20:15:30.980409 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.031748 (+ 51339us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.084149 (+ 52401us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.085748 (+ 1599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2299693}
I20251023 20:15:31.087101 23787 tablet_bootstrap.cc:492] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.087620 23787 ts_tablet_manager.cc:1403] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20251023 20:15:31.088685 23787 raft_consensus.cc:359] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.089864 23787 raft_consensus.cc:385] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.090075 23787 raft_consensus.cc:740] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.079504 25045 ts_tablet_manager.cc:1929] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.090437 25045 log.cc:1199] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/cbacaf2fea24405caaf84d59b625fca3
I20251023 20:15:31.090754 25045 ts_tablet_manager.cc:1950] T cbacaf2fea24405caaf84d59b625fca3 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:31.090987 23787 consensus_queue.cc:260] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.094337 23787 ts_tablet_manager.cc:1434] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251023 20:15:31.095350 23787 tablet_bootstrap.cc:492] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.096400 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet cbacaf2fea24405caaf84d59b625fca3 (table default.Table_8ff1c10b4f2f49dd950540c129481320 [id=4978d9ba6f66482e848efe9eb0c1c7b2]) successfully deleted
I20251023 20:15:31.095647 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 128) took 2415 ms. Trace:
W20251023 20:15:31.097663 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 352812b56f8d4feb818a364d6adfd86d because the tablet was not found. No further retry: Not found: Tablet not found: 352812b56f8d4feb818a364d6adfd86d
I20251023 20:15:31.098960 23790 tablet_bootstrap.cc:492] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.098927 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.099184 23790 ts_tablet_manager.cc:1403] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.143s user 0.000s sys 0.008s
I20251023 20:15:31.099282 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet f4778436aed64b53adb2bab54bcf263b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.097201 24780 rpcz_store.cc:276] 1023 20:15:28.679728 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.679799 (+ 71us) service_pool.cc:225] Handling call
1023 20:15:31.086549 (+2406750us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:31.086560 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.095302 (+ 8742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404861}
W20251023 20:15:31.099982 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6a54470d088748b0be22ab9db5a25f87 because the tablet was not found. No further retry: Not found: Tablet not found: 6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:31.100150 23790 raft_consensus.cc:359] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.100528 23790 raft_consensus.cc:385] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.100719 23786 tablet_bootstrap.cc:492] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.100709 23790 raft_consensus.cc:740] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.101351 23786 ts_tablet_manager.cc:1403] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.063s user 0.000s sys 0.003s
I20251023 20:15:31.101351 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 7540354f8577482785e6341fa9022315 with delete_type TABLET_DATA_DELETED (replaced by tablet e194d03a6aa946fbb2c79d91b196440b) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.101521 23790 consensus_queue.cc:260] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.102094 23786 raft_consensus.cc:359] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.102281 23786 raft_consensus.cc:385] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.102347 23786 raft_consensus.cc:740] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.102571 23786 consensus_queue.cc:260] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.103060 23786 ts_tablet_manager.cc:1434] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20251023 20:15:31.103824 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a9c5b69df4c04365b1ac1457fe8300cc because the tablet was not found. No further retry: Not found: Tablet not found: a9c5b69df4c04365b1ac1457fe8300cc
I20251023 20:15:31.104481 23787 tablet_bootstrap.cc:654] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.104940 23786 tablet_bootstrap.cc:492] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.106751 23786 tablet_bootstrap.cc:654] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.109158 25045 tablet_replica.cc:333] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:31.109409 25045 raft_consensus.cc:2243] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.109664 25045 raft_consensus.cc:2272] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.110234 25045 ts_tablet_manager.cc:1916] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.109766 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 352812b56f8d4feb818a364d6adfd86d (table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]) successfully deleted
I20251023 20:15:31.113778 23218 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 68) took 4433 ms. Trace:
I20251023 20:15:31.115265 25068 ts_tablet_manager.cc:1929] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.115108 23218 rpcz_store.cc:276] 1023 20:15:26.680743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.036305 (+4355562us) service_pool.cc:225] Handling call
1023 20:15:31.037597 (+ 1292us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:31.038102 (+ 505us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:31.104682 (+ 66580us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:31.109796 (+ 5114us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:31.113734 (+ 3938us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.039065 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.039172 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.039180 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:31.039182 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:31.039283 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.039297 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.039301 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.039373 (+ 72us) tablet.cc:699] Row locks acquired
1023 20:15:31.039376 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:31.039414 (+ 38us) write_op.cc:270] Start()
1023 20:15:31.039448 (+ 34us) write_op.cc:276] Timestamp: P: 1761250531039367 usec, L: 0
1023 20:15:31.039452 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.039621 (+ 169us) log.cc:844] Serialized 433 byte log entry
1023 20:15:31.077893 (+ 38272us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.078006 (+ 113us) write_op.cc:301] APPLY: starting
1023 20:15:31.078022 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.078070 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.078107 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.078231 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.078266 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:31.078274 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:31.078365 (+ 91us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.078428 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.078447 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:31.078455 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.078533 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.078537 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":518,"num_ops":2,"prepare.queue_time_us":427,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":660,"raft.queue_time_us":54,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":203,"replication_time_us":38397}]]}
I20251023 20:15:31.115782 23218 master_service.cc:954] ReplaceTablet: received request to replace tablet 76380b341a304d6ba31648fbd1548f05 from {username='slave'} at 127.0.0.1:55414
W20251023 20:15:31.115981 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
W20251023 20:15:31.116961 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a8eaa2167f82475ca2f6a110aba288ec because the tablet was not found. No further retry: Not found: Tablet not found: a8eaa2167f82475ca2f6a110aba288ec
W20251023 20:15:31.117255 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
W20251023 20:15:31.117481 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
W20251023 20:15:31.117697 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5964da70989b4e6cb77fe5d0a52ce9cf because the tablet was not found. No further retry: Not found: Tablet not found: 5964da70989b4e6cb77fe5d0a52ce9cf
I20251023 20:15:31.118856 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 134) took 2419 ms. Trace:
I20251023 20:15:31.120500 25045 ts_tablet_manager.cc:1929] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.121161 25068 log.cc:1199] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:31.121979 25068 ts_tablet_manager.cc:1950] T 6a54470d088748b0be22ab9db5a25f87 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.122815 25045 log.cc:1199] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d9f73477a06542c0b215ae86716cdc4e
I20251023 20:15:31.123301 25045 ts_tablet_manager.cc:1950] T d9f73477a06542c0b215ae86716cdc4e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:31.124331 25068 tablet_replica.cc:333] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.124542 25068 raft_consensus.cc:2243] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.124732 25068 raft_consensus.cc:2272] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.125303 25068 ts_tablet_manager.cc:1916] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:31.117905 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
W20251023 20:15:31.128607 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet edec01a63a3046d9ae5a372fc9068b52 because the tablet was not found. No further retry: Not found: Tablet not found: edec01a63a3046d9ae5a372fc9068b52
I20251023 20:15:31.128800 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet edec01a63a3046d9ae5a372fc9068b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.129855 23790 ts_tablet_manager.cc:1434] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:31.130529 24670 raft_consensus.cc:493] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:31.130690 25045 tablet_replica.cc:333] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:31.130678 24670 raft_consensus.cc:515] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.130883 25045 raft_consensus.cc:2243] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.122869 23786 tablet_bootstrap.cc:492] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.131021 25045 raft_consensus.cc:2272] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.131160 23786 ts_tablet_manager.cc:1403] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.002s
I20251023 20:15:31.131179 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 4beebee06e5341a29c65d0c3cf985ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.131301 23372 tablet_service.cc:1552] Processing DeleteTablet for tablet 52bc660b9da54c5d984381a267e1efa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.131325 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 4ea63540149c4c02abb8ae51209060a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.131465 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.131850 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 83da09ee0d10435ba89a19dc378d34eb with delete_type TABLET_DATA_DELETED (replaced by tablet 25079532547d4175802b84877790662d) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.131469 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.132051 25045 ts_tablet_manager.cc:1916] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.131978 23786 raft_consensus.cc:359] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.132154 23786 raft_consensus.cc:385] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.132284 23786 raft_consensus.cc:740] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.132588 23786 consensus_queue.cc:260] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:31.133133 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7540354f8577482785e6341fa9022315 because the tablet was not found. No further retry: Not found: Tablet not found: 7540354f8577482785e6341fa9022315
I20251023 20:15:31.133301 23786 ts_tablet_manager.cc:1434] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.133625 23786 tablet_bootstrap.cc:492] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:31.134253 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
I20251023 20:15:31.134815 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:31.135010 23790 tablet_bootstrap.cc:492] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.134986 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet cff337a6e728489b99fa8c5d4dfece2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.130353 24780 rpcz_store.cc:276] 1023 20:15:28.681801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.682156 (+ 355us) service_pool.cc:225] Handling call
1023 20:15:31.100354 (+2418198us) ts_tablet_manager.cc:1151] Deleting tablet 1822fdf1f5f44dd4a742aa3c2f8443cf
1023 20:15:31.100366 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.101203 (+ 837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417771}
I20251023 20:15:31.135303 23786 tablet_bootstrap.cc:654] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.138720 23790 tablet_bootstrap.cc:654] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.138877 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a4c8dcbdfcf4a15b8a4194ea04b93a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.139647 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fbad98d3ab24cc5b7726a92fba04cf8 with delete_type TABLET_DATA_DELETED (replaced by tablet cec51c54ea534b1da4b1d170e79d9f1c) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.139947 24670 leader_election.cc:290] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.141389 24297 raft_consensus.cc:493] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:31.142349 25045 ts_tablet_manager.cc:1929] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.142452 25045 log.cc:1199] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/b90810781f724c78935a33d868ffa27d
I20251023 20:15:31.142729 25045 ts_tablet_manager.cc:1950] T b90810781f724c78935a33d868ffa27d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:31.142940 24825 raft_consensus.cc:493] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:31.144560 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a8ba7ab28dfa4ac38e0b58ba2115495c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.145030 23786 tablet_bootstrap.cc:492] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.145301 23786 ts_tablet_manager.cc:1403] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251023 20:15:31.145673 23787 tablet_bootstrap.cc:492] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.145799 23787 ts_tablet_manager.cc:1403] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.005s
I20251023 20:15:31.146041 23786 raft_consensus.cc:359] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.146225 23786 raft_consensus.cc:385] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.146307 23786 raft_consensus.cc:740] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.146479 23787 raft_consensus.cc:359] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.146621 23787 raft_consensus.cc:385] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.146663 23787 raft_consensus.cc:740] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.146582 23786 consensus_queue.cc:260] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.146840 23787 consensus_queue.cc:260] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.147290 23787 ts_tablet_manager.cc:1434] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251023 20:15:31.148097 25068 ts_tablet_manager.cc:1929] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.148272 25068 log.cc:1199] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/600f0110f8bd4b05983d5922d871c266
I20251023 20:15:31.148712 25068 ts_tablet_manager.cc:1950] T 600f0110f8bd4b05983d5922d871c266 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.148721 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 7301d6121c9d4c8dbc642e895aa4e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.148965 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.149134 24297 raft_consensus.cc:515] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:31.149523 23786 ts_tablet_manager.cc:1434] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:31.149554 25068 tablet_replica.cc:333] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.149827 24297 leader_election.cc:290] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.150130 25068 raft_consensus.cc:2243] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.150069 24825 raft_consensus.cc:515] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:31.150614 24825 leader_election.cc:290] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.152005 25068 raft_consensus.cc:2272] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.152585 25068 ts_tablet_manager.cc:1916] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.149850 23786 tablet_bootstrap.cc:492] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.158308 23786 tablet_bootstrap.cc:654] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.159256 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 136) took 2414 ms. Trace:
I20251023 20:15:31.160181 24780 rpcz_store.cc:276] 1023 20:15:28.721505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.721557 (+ 52us) service_pool.cc:225] Handling call
1023 20:15:31.135691 (+2414134us) ts_tablet_manager.cc:1151] Deleting tablet d9f73477a06542c0b215ae86716cdc4e
1023 20:15:31.135792 (+ 101us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.136114 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413881}
I20251023 20:15:31.163668 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 135) took 2441 ms. Trace:
I20251023 20:15:31.166249 23790 tablet_bootstrap.cc:492] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.166433 23790 ts_tablet_manager.cc:1403] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.007s
I20251023 20:15:31.165372 24780 rpcz_store.cc:276] 1023 20:15:28.721416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.721548 (+ 132us) service_pool.cc:225] Handling call
1023 20:15:31.162295 (+2440747us) ts_tablet_manager.cc:1151] Deleting tablet 370a790a053446b2b06314c7db242270
1023 20:15:31.162309 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.163280 (+ 971us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15,"tablet-delete.queue_time_us":2440319}
I20251023 20:15:31.168794 24780 tablet_replica.cc:333] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:31.171686 24780 raft_consensus.cc:2243] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.173326 23790 raft_consensus.cc:359] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.173537 23790 raft_consensus.cc:385] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.173645 23790 raft_consensus.cc:740] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.173887 23790 consensus_queue.cc:260] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.174489 24780 raft_consensus.cc:2272] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.178489 23786 tablet_bootstrap.cc:492] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.178660 23786 ts_tablet_manager.cc:1403] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.004s
I20251023 20:15:31.178507 25068 ts_tablet_manager.cc:1929] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.179629 25068 log.cc:1199] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a636a4cdeb404e9f806d8e166ea57938
I20251023 20:15:31.179374 23786 raft_consensus.cc:359] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.179935 23786 raft_consensus.cc:385] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.179986 25068 ts_tablet_manager.cc:1950] T a636a4cdeb404e9f806d8e166ea57938 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.180040 23786 raft_consensus.cc:740] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.180307 23786 consensus_queue.cc:260] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.180933 25068 tablet_replica.cc:333] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.181157 25068 raft_consensus.cc:2243] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.181340 25068 raft_consensus.cc:2272] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.181859 25068 ts_tablet_manager.cc:1916] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.182497 23787 tablet_bootstrap.cc:492] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.183267 25045 tablet_replica.cc:333] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:31.181160 23786 ts_tablet_manager.cc:1434] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.183482 25045 raft_consensus.cc:2243] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.183636 25045 raft_consensus.cc:2272] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.183724 23786 tablet_bootstrap.cc:492] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.184108 25045 ts_tablet_manager.cc:1916] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.184233 23787 tablet_bootstrap.cc:654] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.185561 23786 tablet_bootstrap.cc:654] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.189761 23790 ts_tablet_manager.cc:1434] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20251023 20:15:31.190088 23787 tablet_bootstrap.cc:492] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.190300 23787 ts_tablet_manager.cc:1403] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20251023 20:15:31.191294 23787 raft_consensus.cc:359] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.191535 23787 raft_consensus.cc:385] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.191196 24780 ts_tablet_manager.cc:1916] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.191640 23787 raft_consensus.cc:740] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.191885 23787 consensus_queue.cc:260] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.192512 23787 ts_tablet_manager.cc:1434] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.192847 23787 tablet_bootstrap.cc:492] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.193794 23790 tablet_bootstrap.cc:492] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.194494 23787 tablet_bootstrap.cc:654] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.141038 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 6e1b4748c0244b258960b1025da47562 (table default.Table_4740fc3fe4ac4927b7f776c980467298 [id=a93a658ff7104725a680f6ee3686e0a4]) successfully deleted
W20251023 20:15:31.197749 23205 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 56799.4us
W20251023 20:15:31.198046 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
W20251023 20:15:31.198285 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
W20251023 20:15:31.198751 23204 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 63474.9us
W20251023 20:15:31.199033 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
W20251023 20:15:31.199286 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
W20251023 20:15:31.199499 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 0858358aabff47a69dc77f4669b225c5 because the tablet was not found. No further retry: Not found: Tablet not found: 0858358aabff47a69dc77f4669b225c5
W20251023 20:15:31.199759 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 58ae6a6e58274b8baf952eaaefde7127 because the tablet was not found. No further retry: Not found: Tablet not found: 58ae6a6e58274b8baf952eaaefde7127
W20251023 20:15:31.199993 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
W20251023 20:15:31.200258 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf because the tablet was not found. No further retry: Not found: Tablet not found: 1e8eb7603c3e46f1b15cdff7a7bbedcf
W20251023 20:15:31.200587 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
I20251023 20:15:31.200598 23786 tablet_bootstrap.cc:492] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.200817 23786 ts_tablet_manager.cc:1403] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
W20251023 20:15:31.201058 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a21ec49194e24723b31fd9c86f1f85e4 because the tablet was not found. No further retry: Not found: Tablet not found: a21ec49194e24723b31fd9c86f1f85e4
W20251023 20:15:31.201527 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
I20251023 20:15:31.201741 23786 raft_consensus.cc:359] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.201946 23786 raft_consensus.cc:385] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.202021 23786 raft_consensus.cc:740] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:31.202209 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
I20251023 20:15:31.202118 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_aeba123341a14c5aba5a1ccca37e5395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:31.202287 23786 consensus_queue.cc:260] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.202503 23212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_7417d01d91d44665a6f021a647db582c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:31.202858 23786 ts_tablet_manager.cc:1434] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251023 20:15:31.202903 23212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7417d01d91d44665a6f021a647db582c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:31.203153 23786 tablet_bootstrap.cc:492] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:31.203495 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 8289ce37fa074891beb584bf2fff9747 because the tablet was not found. No further retry: Not found: Tablet not found: 8289ce37fa074891beb584bf2fff9747
I20251023 20:15:31.168252 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:31.203943 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:31.204231 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:31.204551 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:31.204859 23786 tablet_bootstrap.cc:654] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:31.202878 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeba123341a14c5aba5a1ccca37e5395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:31.206158 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 63745us
I20251023 20:15:31.208150 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:31.208735 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:31.209321 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]). Expected version 1 got 0
I20251023 20:15:31.212337 23787 tablet_bootstrap.cc:492] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.212548 23787 ts_tablet_manager.cc:1403] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20251023 20:15:31.212512 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:31.213531 23787 raft_consensus.cc:359] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.213832 23787 raft_consensus.cc:385] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.213925 23787 raft_consensus.cc:740] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.213642 23790 tablet_bootstrap.cc:654] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:31.213372 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
I20251023 20:15:31.214217 23787 consensus_queue.cc:260] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.214907 23787 ts_tablet_manager.cc:1434] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:31.215514 23787 tablet_bootstrap.cc:492] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.217170 23787 tablet_bootstrap.cc:654] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.219903 25068 ts_tablet_manager.cc:1929] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:31.216938 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
W20251023 20:15:31.221378 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ae388da9e4f94ba5af82a67088ab7097 because the tablet was not found. No further retry: Not found: Tablet not found: ae388da9e4f94ba5af82a67088ab7097
I20251023 20:15:31.221642 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
W20251023 20:15:31.228318 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 46f700ea3db04a4bad539c943e850a44 because the tablet was not found. No further retry: Not found: Tablet not found: 46f700ea3db04a4bad539c943e850a44
I20251023 20:15:31.230600 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
W20251023 20:15:31.231484 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
I20251023 20:15:31.231832 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:31.234129 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:31.233558 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet d9f73477a06542c0b215ae86716cdc4e (table default.Table_79be15cb4350452b90277004ec803c8a [id=6604e04478d9438e990e19240dd10e73]) successfully deleted
W20251023 20:15:31.236783 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
I20251023 20:15:31.230126 23786 tablet_bootstrap.cc:492] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.237265 25068 log.cc:1199] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/af899c01b4af46d39d27f984cc3d5940
I20251023 20:15:31.237327 23786 ts_tablet_manager.cc:1403] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.005s
I20251023 20:15:31.237708 25068 ts_tablet_manager.cc:1950] T af899c01b4af46d39d27f984cc3d5940 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.238163 23786 raft_consensus.cc:359] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.238348 23786 raft_consensus.cc:385] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.238427 23786 raft_consensus.cc:740] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.238616 25068 tablet_replica.cc:333] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.238657 23786 consensus_queue.cc:260] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.238842 25068 raft_consensus.cc:2243] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.238988 25068 raft_consensus.cc:2272] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.239152 23786 ts_tablet_manager.cc:1434] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.239533 25068 ts_tablet_manager.cc:1916] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.240494 23787 tablet_bootstrap.cc:492] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.240695 23787 ts_tablet_manager.cc:1403] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
W20251023 20:15:31.240799 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 6e1b4748c0244b258960b1025da47562 because the tablet was not found. No further retry: Not found: Tablet not found: 6e1b4748c0244b258960b1025da47562
I20251023 20:15:31.241554 23787 raft_consensus.cc:359] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.241797 23787 raft_consensus.cc:385] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.241901 23787 raft_consensus.cc:740] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.242192 23787 consensus_queue.cc:260] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.241868 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:31.242852 23787 ts_tablet_manager.cc:1434] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:31.243201 23787 tablet_bootstrap.cc:492] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:31.243530 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 8317a54a5f7b4afe8da9d336268efe43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8317a54a5f7b4afe8da9d336268efe43 already in progress: creating tablet
I20251023 20:15:31.243690 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:31.244975 23787 tablet_bootstrap.cc:654] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.239667 23786 tablet_bootstrap.cc:492] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.246881 23786 tablet_bootstrap.cc:654] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.247807 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:31.248160 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:31.248319 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:31.248590 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:31.248956 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:31.249194 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:31.249440 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
W20251023 20:15:31.250967 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a2dbb7997d0f4e6bac48cfb196f5ccb6 because the tablet was not found. No further retry: Not found: Tablet not found: a2dbb7997d0f4e6bac48cfb196f5ccb6
W20251023 20:15:31.255151 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e19859a65a144598e1612be89daa3a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e19859a65a144598e1612be89daa3a1 already in progress: creating tablet
W20251023 20:15:31.256911 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 40e0301fb5974ec5b6e8291230a736c2 because the tablet was not found. No further retry: Not found: Tablet not found: 40e0301fb5974ec5b6e8291230a736c2
W20251023 20:15:31.258662 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 25079532547d4175802b84877790662d because the tablet was not found. No further retry: Not found: Tablet not found: 25079532547d4175802b84877790662d
I20251023 20:15:31.260849 23786 tablet_bootstrap.cc:492] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.261107 23786 ts_tablet_manager.cc:1403] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.001s
W20251023 20:15:31.260592 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ac55e14745534a02a2f6a422cab6b149 because the tablet was not found. No further retry: Not found: Tablet not found: ac55e14745534a02a2f6a422cab6b149
I20251023 20:15:31.261960 23786 raft_consensus.cc:359] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.262167 23786 raft_consensus.cc:385] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.262239 23786 raft_consensus.cc:740] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.262501 23786 consensus_queue.cc:260] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.263078 23786 ts_tablet_manager.cc:1434] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:31.263412 23786 tablet_bootstrap.cc:492] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:31.263554 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fe922c438a894a7aa04699b2d2897a28 because the tablet was not found. No further retry: Not found: Tablet not found: fe922c438a894a7aa04699b2d2897a28
I20251023 20:15:31.266224 23786 tablet_bootstrap.cc:654] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.266345 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet b90810781f724c78935a33d868ffa27d (table default.Table_fab9daaa8e644d2a9681e2229696c2bb [id=2df3ca33fcf54a978b31fc24b9d21ced]) successfully deleted
I20251023 20:15:31.267993 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
W20251023 20:15:31.269125 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a27f192792cf41288d69ca15ab538472 because the tablet was not found. No further retry: Not found: Tablet not found: a27f192792cf41288d69ca15ab538472
I20251023 20:15:31.269955 25061 raft_consensus.cc:493] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.270095 25061 raft_consensus.cc:515] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.270733 25061 leader_election.cc:290] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.271433 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a570f339890946b7ab20c15de68462d5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.271755 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a570f339890946b7ab20c15de68462d5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.272362 23604 leader_election.cc:343] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:31.272742 23605 leader_election.cc:343] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.272840 23605 leader_election.cc:304] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:31.272014 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e5368813164949ff80e27bf59dcc931a because the tablet was not found. No further retry: Not found: Tablet not found: e5368813164949ff80e27bf59dcc931a
I20251023 20:15:31.273209 25061 raft_consensus.cc:2749] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.273365 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:31.275830 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
W20251023 20:15:31.275753 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 15ced74d0dd74e5195ed98358957d0dd because the tablet was not found. No further retry: Not found: Tablet not found: 15ced74d0dd74e5195ed98358957d0dd
I20251023 20:15:31.276840 23787 tablet_bootstrap.cc:492] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.277222 23787 ts_tablet_manager.cc:1403] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
I20251023 20:15:31.278097 23787 raft_consensus.cc:359] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.278357 23787 raft_consensus.cc:385] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.278448 23787 raft_consensus.cc:740] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:31.278008 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2532c8567db14901b383f42e5f04b4c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2532c8567db14901b383f42e5f04b4c2
I20251023 20:15:31.278748 23787 consensus_queue.cc:260] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.281577 23787 ts_tablet_manager.cc:1434] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:31.281968 23787 tablet_bootstrap.cc:492] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.283725 23787 tablet_bootstrap.cc:654] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.283955 23212 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 72) took 4505 ms. Trace:
I20251023 20:15:31.284154 23212 rpcz_store.cc:276] 1023 20:15:26.778912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.086281 (+4307369us) service_pool.cc:225] Handling call
1023 20:15:31.202454 (+116173us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a06027a3 00007f8997106de6 00007f8997106b15 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:31.203009 (+ 555us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:31.203475 (+ 466us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:31.283703 (+ 80228us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:31.283813 (+ 110us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:31.283908 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.204051 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.204158 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.204165 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:31.204167 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:31.204281 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.204295 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.204299 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.204373 (+ 74us) tablet.cc:699] Row locks acquired
1023 20:15:31.204375 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:31.204417 (+ 42us) write_op.cc:270] Start()
1023 20:15:31.204452 (+ 35us) write_op.cc:276] Timestamp: P: 1761250531204391 usec, L: 0
1023 20:15:31.204456 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.204667 (+ 211us) log.cc:844] Serialized 433 byte log entry
1023 20:15:31.283052 (+ 78385us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.283216 (+ 164us) write_op.cc:301] APPLY: starting
1023 20:15:31.283232 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.283280 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.283312 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.283414 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.283449 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:31.283455 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:31.283531 (+ 76us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.283594 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.283612 (+ 18us) write_op.cc:454] Released schema lock
1023 20:15:31.283619 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.283696 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.283700 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":266617728,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":472,"apply.run_wall_time_us":471,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":689,"replication_time_us":78562}]]}
I20251023 20:15:31.284747 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 71) took 4519 ms. Trace:
I20251023 20:15:31.284866 23212 master_service.cc:954] ReplaceTablet: received request to replace tablet 5964da70989b4e6cb77fe5d0a52ce9cf from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:31.285285 23212 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 74) took 4342 ms. Trace:
I20251023 20:15:31.285380 23212 rpcz_store.cc:276] 1023 20:15:26.943119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.284789 (+4341670us) service_pool.cc:225] Handling call
1023 20:15:31.285232 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:31.285655 23212 master_service.cc:954] ReplaceTablet: received request to replace tablet 7f2a4f1d52984af58d6286671389b6a9 from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:31.286556 23786 tablet_bootstrap.cc:492] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.286754 23786 ts_tablet_manager.cc:1403] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20251023 20:15:31.284873 23210 rpcz_store.cc:276] 1023 20:15:26.765313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.084974 (+4319661us) service_pool.cc:225] Handling call
1023 20:15:31.202061 (+117087us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a06027a3 00007f8997106de6 00007f8997106b15 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:31.205114 (+ 3053us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:31.205560 (+ 446us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:31.284519 (+ 78959us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:31.284608 (+ 89us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:31.284701 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.206250 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.206347 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.206353 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:31.206355 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:31.206433 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.206442 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.206447 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.206514 (+ 67us) tablet.cc:699] Row locks acquired
1023 20:15:31.206516 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:31.206633 (+ 117us) write_op.cc:270] Start()
1023 20:15:31.206670 (+ 37us) write_op.cc:276] Timestamp: P: 1761250531206600 usec, L: 0
1023 20:15:31.206674 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.206853 (+ 179us) log.cc:844] Serialized 433 byte log entry
1023 20:15:31.283120 (+ 76267us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.283775 (+ 655us) write_op.cc:301] APPLY: starting
1023 20:15:31.283785 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.283818 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.283848 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.283968 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.283995 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:31.284001 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:31.284064 (+ 63us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.284107 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.284124 (+ 17us) write_op.cc:454] Released schema lock
1023 20:15:31.284130 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.284199 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.284203 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":268573824,"child_traces":[["op",{"apply.queue_time_us":622,"apply.run_cpu_time_us":417,"apply.run_wall_time_us":417,"num_ops":2,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":671,"prepare.run_wall_time_us":671,"replication_time_us":76430}]]}
I20251023 20:15:31.287081 23210 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_532d7e6d0eec4f09bcdf525c353fe0a8" } modify_external_catalogs: true
I20251023 20:15:31.287268 23210 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_532d7e6d0eec4f09bcdf525c353fe0a8" } modify_external_catalogs: true
I20251023 20:15:31.287487 23786 raft_consensus.cc:359] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.287647 23786 raft_consensus.cc:385] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.287717 23786 raft_consensus.cc:740] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.287923 23786 consensus_queue.cc:260] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.289556 24822 raft_consensus.cc:493] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.289781 24822 raft_consensus.cc:515] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.290450 24822 leader_election.cc:290] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:31.289266 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d951134f6773410386bc57f7f90310c3 because the tablet was not found. No further retry: Not found: Tablet not found: d951134f6773410386bc57f7f90310c3
W20251023 20:15:31.291767 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 79e3d422a66f432789c0cf4a00e57af2 because the tablet was not found. No further retry: Not found: Tablet not found: 79e3d422a66f432789c0cf4a00e57af2
W20251023 20:15:31.293185 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 66a2f3adb5d347a0adc198cd1ef93304 because the tablet was not found. No further retry: Not found: Tablet not found: 66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:31.293138 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
W20251023 20:15:31.294296 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5346a221c78342359eb6e418bd12cd24 because the tablet was not found. No further retry: Not found: Tablet not found: 5346a221c78342359eb6e418bd12cd24
I20251023 20:15:31.295872 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
W20251023 20:15:31.295840 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 0438593fbbdf47cebfb64121acad104a because the tablet was not found. No further retry: Not found: Tablet not found: 0438593fbbdf47cebfb64121acad104a
I20251023 20:15:31.296911 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
W20251023 20:15:31.297552 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 252b5480222f49439052c86a4115cc9b because the tablet was not found. No further retry: Not found: Tablet not found: 252b5480222f49439052c86a4115cc9b
I20251023 20:15:31.298806 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
W20251023 20:15:31.299216 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet f908f086946949829cac15418a584730 because the tablet was not found. No further retry: Not found: Tablet not found: f908f086946949829cac15418a584730
I20251023 20:15:31.300257 23786 ts_tablet_manager.cc:1434] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:31.300316 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:31.300665 23786 tablet_bootstrap.cc:492] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:31.300264 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 because the tablet was not found. No further retry: Not found: Tablet not found: 283166bbd8004ce3bca9f2ecbdbcbf48
I20251023 20:15:31.302299 23786 tablet_bootstrap.cc:654] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.301985 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:31.305395 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
W20251023 20:15:31.307933 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9f73477a06542c0b215ae86716cdc4e already in progress: creating tablet
W20251023 20:15:31.308224 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
W20251023 20:15:31.309932 23333 leader_election.cc:343] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 66a2f3adb5d347a0adc198cd1ef93304
W20251023 20:15:31.310410 23333 leader_election.cc:343] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 4b6c3e7169154e3497dd3570dca0d745
W20251023 20:15:31.310715 23333 leader_election.cc:343] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 951b2276d3324c2d999c8d7fb97767bc
W20251023 20:15:31.312361 23333 leader_election.cc:343] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:31.312569 23333 leader_election.cc:304] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.312849 25092 raft_consensus.cc:2749] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:31.312583 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:31.314298 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 6a54470d088748b0be22ab9db5a25f87 (table default.Table_bd447f978d1d4b8bad839b915907db66 [id=a48d0e1e007c40d59766c7d0ef778956]) successfully deleted
I20251023 20:15:31.315126 23786 tablet_bootstrap.cc:492] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.315346 23786 ts_tablet_manager.cc:1403] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20251023 20:15:31.315141 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:31.316102 23786 raft_consensus.cc:359] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.316345 23786 raft_consensus.cc:385] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.316428 23786 raft_consensus.cc:740] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.316685 23786 consensus_queue.cc:260] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:31.316674 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d82c92e863b2470699cc822b8cb8c14d because the tablet was not found. No further retry: Not found: Tablet not found: d82c92e863b2470699cc822b8cb8c14d
I20251023 20:15:31.317334 23786 ts_tablet_manager.cc:1434] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.317960 23786 tablet_bootstrap.cc:492] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.320045 23786 tablet_bootstrap.cc:654] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:31.319526 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet cec51c54ea534b1da4b1d170e79d9f1c because the tablet was not found. No further retry: Not found: Tablet not found: cec51c54ea534b1da4b1d170e79d9f1c
W20251023 20:15:31.321527 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 6480cdc1afda4671a346044f232b083e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6480cdc1afda4671a346044f232b083e already in progress: creating tablet
I20251023 20:15:31.321918 23787 tablet_bootstrap.cc:492] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.322047 23787 ts_tablet_manager.cc:1403] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20251023 20:15:31.322233 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_d22911da756748fb8939006df660e885 [id=78fe983663c94eabab93ee902bde6998]). Expected version 5 got 4
I20251023 20:15:31.322921 23787 raft_consensus.cc:359] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.323097 23787 raft_consensus.cc:385] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.323159 23787 raft_consensus.cc:740] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.323414 23787 consensus_queue.cc:260] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.323612 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 600f0110f8bd4b05983d5922d871c266 (table default.Table_f7824cd559744860a83775305e47389c [id=4b9ecb1c28154b86bc2b6da54948416b]) successfully deleted
I20251023 20:15:31.324301 23787 ts_tablet_manager.cc:1434] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:31.324663 23787 tablet_bootstrap.cc:492] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.325119 25045 ts_tablet_manager.cc:1929] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.326381 23790 tablet_bootstrap.cc:492] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.326606 23790 ts_tablet_manager.cc:1403] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.133s user 0.008s sys 0.008s
I20251023 20:15:31.328312 23790 raft_consensus.cc:359] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.328599 23790 raft_consensus.cc:385] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.328727 23790 raft_consensus.cc:740] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.328999 23790 consensus_queue.cc:260] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.329591 23790 ts_tablet_manager.cc:1434] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:31.330179 23790 tablet_bootstrap.cc:492] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.325588 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet a636a4cdeb404e9f806d8e166ea57938 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]) successfully deleted
I20251023 20:15:31.331391 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:31.332581 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet af899c01b4af46d39d27f984cc3d5940 (table default.Table_6d990f17621b44f78107cef2a0994e8e [id=8d5fb738eb684e27860e03c12802a399]) successfully deleted
I20251023 20:15:31.336136 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:31.336779 24830 raft_consensus.cc:493] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.336927 24830 raft_consensus.cc:515] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.337553 24830 leader_election.cc:290] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.338227 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
W20251023 20:15:31.340119 23470 leader_election.cc:343] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 58ae6a6e58274b8baf952eaaefde7127
I20251023 20:15:31.340794 23790 tablet_bootstrap.cc:654] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.340636 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:31.341938 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:31.342350 23786 tablet_bootstrap.cc:492] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.342734 23786 ts_tablet_manager.cc:1403] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20251023 20:15:31.342942 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:31.343439 23786 raft_consensus.cc:359] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.343667 23786 raft_consensus.cc:385] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.343756 23786 raft_consensus.cc:740] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.344008 23786 consensus_queue.cc:260] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.344635 23786 ts_tablet_manager.cc:1434] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.345002 23786 tablet_bootstrap.cc:492] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:31.345150 23333 leader_election.cc:343] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 4b6c3e7169154e3497dd3570dca0d745
I20251023 20:15:31.344616 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:31.345347 23333 leader_election.cc:304] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.345671 25092 raft_consensus.cc:2749] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:31.345814 23333 leader_election.cc:343] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:31.345923 23333 leader_election.cc:304] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.346117 25092 raft_consensus.cc:2749] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:31.346498 23786 tablet_bootstrap.cc:654] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.346426 25092 consensus_queue.cc:1048] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:31.347838 23528 raft_consensus.cc:3060] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.352579 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:31.354032 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:31.354900 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:31.356132 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:31.357499 23216 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:31.359526 24830 raft_consensus.cc:493] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.359688 24830 raft_consensus.cc:515] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.360323 24830 leader_election.cc:290] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.337967 23787 tablet_bootstrap.cc:654] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.372479 25045 log.cc:1199] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/bb475e303d07491bbc19dc3f628bceb7
I20251023 20:15:31.372865 25045 ts_tablet_manager.cc:1950] T bb475e303d07491bbc19dc3f628bceb7 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:31.347182 25061 raft_consensus.cc:493] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.373206 25061 raft_consensus.cc:515] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.373885 25061 leader_election.cc:290] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.361500 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:31.375437 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f7dffd94f3442ea20045a512d7bd65" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.375548 23670 raft_consensus.cc:2393] T 98a4553c1f014ef3ac975bf7a8076334 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
W20251023 20:15:31.375984 23605 leader_election.cc:343] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.376423 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f7dffd94f3442ea20045a512d7bd65" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:31.377158 23604 leader_election.cc:343] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.377259 23604 leader_election.cc:304] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.378111 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet bb475e303d07491bbc19dc3f628bceb7 (table default.Table_0d31043ac68d4a0394d7ba5b60d539c8 [id=0a9d3a3303d945f199d265637a859772]) successfully deleted
W20251023 20:15:31.378398 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1a64a7c2e1f242e1b54243489622bae5 because the tablet was not found. No further retry: Not found: Tablet not found: 1a64a7c2e1f242e1b54243489622bae5
W20251023 20:15:31.378608 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d2ae19cc80194363b8a9c96fbafb4ebe because the tablet was not found. No further retry: Not found: Tablet not found: d2ae19cc80194363b8a9c96fbafb4ebe
I20251023 20:15:31.378737 25061 raft_consensus.cc:2749] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:31.378929 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 4b6c3e7169154e3497dd3570dca0d745 because the tablet was not found. No further retry: Not found: Tablet not found: 4b6c3e7169154e3497dd3570dca0d745
W20251023 20:15:31.379213 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9a4f012469fa4e8e827e485287ad76dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a4f012469fa4e8e827e485287ad76dd already in progress: creating tablet
W20251023 20:15:31.379491 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5e73ae95c73945f081c298f49d47c904 because the tablet was not found. No further retry: Not found: Tablet not found: 5e73ae95c73945f081c298f49d47c904
W20251023 20:15:31.379920 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 86b9f4c5d9e0469c87071a33722ce3eb because the tablet was not found. No further retry: Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb
W20251023 20:15:31.380224 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1ae910ac41fd4e349f159ab27e44375a because the tablet was not found. No further retry: Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
W20251023 20:15:31.380712 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 951b2276d3324c2d999c8d7fb97767bc because the tablet was not found. No further retry: Not found: Tablet not found: 951b2276d3324c2d999c8d7fb97767bc
W20251023 20:15:31.381050 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
W20251023 20:15:31.381330 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 994a555b24e4432a84601c54d422aa5d because the tablet was not found. No further retry: Not found: Tablet not found: 994a555b24e4432a84601c54d422aa5d
W20251023 20:15:31.381623 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 377de4ea0bd6489284fac6b0fab52c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377de4ea0bd6489284fac6b0fab52c8c already in progress: creating tablet
I20251023 20:15:31.381659 23787 tablet_bootstrap.cc:492] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.381778 23787 ts_tablet_manager.cc:1403] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
W20251023 20:15:31.381978 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 419e41d7a37f4b888cb242b089c28790 because the tablet was not found. No further retry: Not found: Tablet not found: 419e41d7a37f4b888cb242b089c28790
W20251023 20:15:31.382259 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d62dbf044acd4459b26d2e7e50f6206e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d62dbf044acd4459b26d2e7e50f6206e already in progress: creating tablet
I20251023 20:15:31.382548 23787 raft_consensus.cc:359] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.382740 23787 raft_consensus.cc:385] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.382814 23787 raft_consensus.cc:740] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.383040 23787 consensus_queue.cc:260] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.383523 23787 ts_tablet_manager.cc:1434] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:31.386045 23787 tablet_bootstrap.cc:492] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.389935 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c58a406d564b048fb16a85aa0b754b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.390208 23528 raft_consensus.cc:2468] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:31.393428 23787 tablet_bootstrap.cc:654] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.399149 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c58a406d564b048fb16a85aa0b754b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:31.399448 23668 raft_consensus.cc:2468] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:31.405243 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ae6a6e58274b8baf952eaaefde7127" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.405413 25040 raft_consensus.cc:493] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:31.405476 23397 raft_consensus.cc:2468] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:31.405529 25040 raft_consensus.cc:515] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:31.405949 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a4553c1f014ef3ac975bf7a8076334" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.406104 25040 leader_election.cc:290] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.407331 23212 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 7f2a4f1d52984af58d6286671389b6a9
I20251023 20:15:31.407891 23212 catalog_manager.cc:6719] ReplaceTablet: tablet 7f2a4f1d52984af58d6286671389b6a9 deleted and replaced by tablet 9047d641796b4f20b0de82b277ce877c
I20251023 20:15:31.408008 23212 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 75) took 4402 ms. Trace:
I20251023 20:15:31.408106 23212 rpcz_store.cc:276] 1023 20:15:27.005350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.285576 (+4280226us) service_pool.cc:225] Handling call
1023 20:15:31.407979 (+122403us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.397721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.397846 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.397852 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:31.397855 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:31.397963 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.397976 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.397981 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.398055 (+ 74us) tablet.cc:699] Row locks acquired
1023 20:15:31.398057 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:31.398097 (+ 40us) write_op.cc:270] Start()
1023 20:15:31.398129 (+ 32us) write_op.cc:276] Timestamp: P: 1761250531398057 usec, L: 0
1023 20:15:31.398133 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.398340 (+ 207us) log.cc:844] Serialized 631 byte log entry
1023 20:15:31.402666 (+ 4326us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.404710 (+ 2044us) write_op.cc:301] APPLY: starting
1023 20:15:31.404727 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.404785 (+ 58us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.404818 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.404943 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.404994 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:31.405001 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:31.405103 (+ 102us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.405192 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.405213 (+ 21us) write_op.cc:454] Released schema lock
1023 20:15:31.405220 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.405302 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.405306 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1980,"apply.run_cpu_time_us":584,"apply.run_wall_time_us":584,"num_ops":2,"prepare.queue_time_us":338,"prepare.run_cpu_time_us":734,"prepare.run_wall_time_us":734,"raft.queue_time_us":441,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":268,"replication_time_us":4498}]]}
I20251023 20:15:31.410051 23471 leader_election.cc:304] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:31.410544 24830 raft_consensus.cc:2804] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.410622 24830 raft_consensus.cc:493] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.410682 24830 raft_consensus.cc:3060] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.414147 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9047d641796b4f20b0de82b277ce877c) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:31.414754 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9047d641796b4f20b0de82b277ce877c) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:31.415062 24830 raft_consensus.cc:515] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:31.416774 23470 leader_election.cc:343] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 58ae6a6e58274b8baf952eaaefde7127
I20251023 20:15:31.417073 24830 leader_election.cc:290] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:31.418474 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
W20251023 20:15:31.421761 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 2fa6a838b1bd409ba2a4e2074576ee23 because the tablet was not found. No further retry: Not found: Tablet not found: 2fa6a838b1bd409ba2a4e2074576ee23
I20251023 20:15:31.422461 23471 leader_election.cc:304] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:31.423137 24830 raft_consensus.cc:3060] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.425717 24830 raft_consensus.cc:2749] T 98a4553c1f014ef3ac975bf7a8076334 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.426875 25061 raft_consensus.cc:493] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.426868 25095 consensus_queue.cc:1048] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:31.427006 25061 raft_consensus.cc:515] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.427579 25061 leader_election.cc:290] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.428615 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "975fc4848c8f4db4bfd2dad9ac620d83" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.429250 23605 leader_election.cc:343] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.429565 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "975fc4848c8f4db4bfd2dad9ac620d83" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:31.430100 23604 leader_election.cc:343] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.430209 23604 leader_election.cc:304] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.430706 25061 raft_consensus.cc:2749] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.430840 25068 ts_tablet_manager.cc:1929] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251023 20:15:31.431449 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 925e53d9ea6f4c7baa38551eb917828d because the tablet was not found. No further retry: Not found: Tablet not found: 925e53d9ea6f4c7baa38551eb917828d
W20251023 20:15:31.432186 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a708cd011802458aa0e2991fd4b4a2d3 because the tablet was not found. No further retry: Not found: Tablet not found: a708cd011802458aa0e2991fd4b4a2d3
I20251023 20:15:31.432401 25068 log.cc:1199] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/f103bece2fa240c19c23ac51046c13ed
W20251023 20:15:31.432866 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
I20251023 20:15:31.433354 25092 raft_consensus.cc:493] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.433492 25092 raft_consensus.cc:515] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
W20251023 20:15:31.435160 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
I20251023 20:15:31.426736 23333 leader_election.cc:304] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:31.435843 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because the tablet was not found. No further retry: Not found: Tablet not found: 9ed0aa7e36e64843ad6b2ad4b415832a
I20251023 20:15:31.432929 25068 ts_tablet_manager.cc:1950] T f103bece2fa240c19c23ac51046c13ed P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.437486 24821 raft_consensus.cc:2804] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.437582 24821 raft_consensus.cc:493] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.437651 24821 raft_consensus.cc:3060] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:31.441761 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet cb915d114fc0421999d079c23cad8eb5 because the tablet was not found. No further retry: Not found: Tablet not found: cb915d114fc0421999d079c23cad8eb5
W20251023 20:15:31.442181 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
W20251023 20:15:31.442489 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 7f2a4f1d52984af58d6286671389b6a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f2a4f1d52984af58d6286671389b6a9 already in progress: creating tablet
I20251023 20:15:31.442803 25040 consensus_queue.cc:1048] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:31.443207 23786 tablet_bootstrap.cc:492] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.443400 23786 ts_tablet_manager.cc:1403] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.098s user 0.017s sys 0.026s
W20251023 20:15:31.443898 23333 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb
I20251023 20:15:31.444172 25068 tablet_replica.cc:333] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.444348 25068 raft_consensus.cc:2243] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.444491 25068 raft_consensus.cc:2272] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.444392 23786 raft_consensus.cc:359] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.444728 23786 raft_consensus.cc:385] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.444706 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet f103bece2fa240c19c23ac51046c13ed (table default.Table_c7fa864846ef4acbba390d7f493a8b58 [id=071ca3495096421eb75fea3fb71890ad]) successfully deleted
I20251023 20:15:31.444824 23786 raft_consensus.cc:740] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.444999 25068 ts_tablet_manager.cc:1916] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.445199 23786 consensus_queue.cc:260] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.445861 23786 ts_tablet_manager.cc:1434] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.445844 24822 consensus_queue.cc:1048] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:31.446156 23786 tablet_bootstrap.cc:492] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.450409 23786 tablet_bootstrap.cc:654] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.453897 25061 raft_consensus.cc:493] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.454063 25061 raft_consensus.cc:515] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.454708 25061 leader_election.cc:290] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.455507 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.455890 23790 tablet_bootstrap.cc:492] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.456199 23790 ts_tablet_manager.cc:1403] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.126s user 0.006s sys 0.000s
W20251023 20:15:31.456910 23605 leader_election.cc:343] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.457288 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:31.457901 23604 leader_election.cc:343] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.457998 23604 leader_election.cc:304] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.457763 23790 raft_consensus.cc:359] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.458123 23790 raft_consensus.cc:385] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.458318 23790 raft_consensus.cc:740] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.458568 25061 raft_consensus.cc:2749] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.458767 23790 consensus_queue.cc:260] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.459280 23786 tablet_bootstrap.cc:492] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.459427 23786 ts_tablet_manager.cc:1403] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20251023 20:15:31.459779 23790 ts_tablet_manager.cc:1434] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:31.460156 23786 raft_consensus.cc:359] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.460346 23786 raft_consensus.cc:385] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.460407 23786 raft_consensus.cc:740] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.460626 23786 consensus_queue.cc:260] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.461189 23786 ts_tablet_manager.cc:1434] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.461457 23786 tablet_bootstrap.cc:492] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.434027 25092 leader_election.cc:290] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:31.466367 23786 tablet_bootstrap.cc:654] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.466876 23787 tablet_bootstrap.cc:492] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.467012 23787 ts_tablet_manager.cc:1403] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.003s
I20251023 20:15:31.467514 24821 raft_consensus.cc:515] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.467672 23787 raft_consensus.cc:359] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.467841 23787 raft_consensus.cc:385] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.467893 23787 raft_consensus.cc:740] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.468134 24821 leader_election.cc:290] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.468102 23787 consensus_queue.cc:260] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.468597 23787 ts_tablet_manager.cc:1434] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:31.469751 23790 tablet_bootstrap.cc:492] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.469972 25061 raft_consensus.cc:493] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.470082 25061 raft_consensus.cc:515] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.470670 25061 leader_election.cc:290] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.473313 23787 tablet_bootstrap.cc:492] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.473308 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92c50c4845524163834861438291a3ec" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.473939 23605 leader_election.cc:343] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.475703 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92c50c4845524163834861438291a3ec" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:31.476378 23604 leader_election.cc:343] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.476511 23604 leader_election.cc:304] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.477061 25061 raft_consensus.cc:2749] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.477707 23790 tablet_bootstrap.cc:654] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.485586 23787 tablet_bootstrap.cc:654] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.493703 24830 raft_consensus.cc:493] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.493881 24830 raft_consensus.cc:515] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:31.494045 23333 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb
I20251023 20:15:31.494164 23333 leader_election.cc:304] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.494498 24830 leader_election.cc:290] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:31.495396 23470 leader_election.cc:343] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 419e41d7a37f4b888cb242b089c28790
I20251023 20:15:31.498005 25068 ts_tablet_manager.cc:1929] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.498206 25068 log.cc:1199] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d9f422575c7c4251879130501ab721fd
I20251023 20:15:31.498618 25068 ts_tablet_manager.cc:1950] T d9f422575c7c4251879130501ab721fd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.499727 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d9f422575c7c4251879130501ab721fd (table default.Table_c01f9e9030974e2e8b43da4e68ef0145 [id=12899265e5544ac4837f4916f8bab077]) successfully deleted
I20251023 20:15:31.500313 24822 raft_consensus.cc:2749] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:31.500521 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c97e2e0f839a4ef389fe100b6a81301e because the tablet was not found. No further retry: Not found: Tablet not found: c97e2e0f839a4ef389fe100b6a81301e
I20251023 20:15:31.500840 25068 tablet_replica.cc:333] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.501029 25068 raft_consensus.cc:2243] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.501219 25068 raft_consensus.cc:2272] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.502827 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:31.512732 23787 tablet_bootstrap.cc:492] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.512951 23787 ts_tablet_manager.cc:1403] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20251023 20:15:31.513924 23787 raft_consensus.cc:359] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.514142 23787 raft_consensus.cc:385] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.514225 23787 raft_consensus.cc:740] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.514453 23787 consensus_queue.cc:260] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.515014 23787 ts_tablet_manager.cc:1434] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:31.516224 23790 tablet_bootstrap.cc:492] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.516383 23790 ts_tablet_manager.cc:1403] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20251023 20:15:31.517132 23790 raft_consensus.cc:359] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.517319 23790 raft_consensus.cc:385] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.517371 23790 raft_consensus.cc:740] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.517793 23790 consensus_queue.cc:260] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.522130 23790 ts_tablet_manager.cc:1434] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:31.522428 23790 tablet_bootstrap.cc:492] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.524238 23790 tablet_bootstrap.cc:654] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.525696 25068 ts_tablet_manager.cc:1916] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.531510 25068 ts_tablet_manager.cc:1929] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.531594 24780 ts_tablet_manager.cc:1929] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:31.531983 25068 log.cc:1199] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/cbacaf2fea24405caaf84d59b625fca3
I20251023 20:15:31.532387 25068 ts_tablet_manager.cc:1950] T cbacaf2fea24405caaf84d59b625fca3 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.532235 24780 log.cc:1199] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/0438593fbbdf47cebfb64121acad104a
I20251023 20:15:31.532486 23786 tablet_bootstrap.cc:492] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.532966 23786 ts_tablet_manager.cc:1403] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.006s
I20251023 20:15:31.533809 23786 raft_consensus.cc:359] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.534047 23786 raft_consensus.cc:385] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.534147 23786 raft_consensus.cc:740] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.534217 25040 raft_consensus.cc:493] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.534426 23786 consensus_queue.cc:260] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.534420 25040 raft_consensus.cc:515] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.534979 23786 ts_tablet_manager.cc:1434] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:31.535077 25040 leader_election.cc:290] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.537659 25040 raft_consensus.cc:493] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.537871 25040 raft_consensus.cc:515] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.537904 24780 ts_tablet_manager.cc:1950] T 0438593fbbdf47cebfb64121acad104a P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:31.538617 25040 leader_election.cc:290] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.538699 23786 tablet_bootstrap.cc:492] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.541329 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 137) took 2782 ms. Trace:
I20251023 20:15:31.543241 23787 tablet_bootstrap.cc:492] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.542407 24780 rpcz_store.cc:276] 1023 20:15:28.758981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.759245 (+ 264us) service_pool.cc:225] Handling call
1023 20:15:31.168279 (+2409034us) ts_tablet_manager.cc:1151] Deleting tablet 0438593fbbdf47cebfb64121acad104a
1023 20:15:31.168294 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.498766 (+330472us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.531015 (+ 32249us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.540960 (+ 9945us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408589}
I20251023 20:15:31.545830 23787 tablet_bootstrap.cc:654] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.546123 23790 tablet_bootstrap.cc:492] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.546154 24780 tablet_replica.cc:333] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:31.546257 23790 ts_tablet_manager.cc:1403] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20251023 20:15:31.547535 23790 raft_consensus.cc:359] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.547713 23790 raft_consensus.cc:385] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.547772 23790 raft_consensus.cc:740] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.548075 23790 consensus_queue.cc:260] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.548470 23790 ts_tablet_manager.cc:1434] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.548772 23790 tablet_bootstrap.cc:492] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.549240 24780 raft_consensus.cc:2243] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:31.550457 23790 tablet_bootstrap.cc:654] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.552760 25068 tablet_replica.cc:333] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.552649 24780 pending_rounds.cc:70] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:31.553066 25068 raft_consensus.cc:2243] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.553349 25068 raft_consensus.cc:2272] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.553581 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet cbacaf2fea24405caaf84d59b625fca3 (table default.Table_8ff1c10b4f2f49dd950540c129481320 [id=4978d9ba6f66482e848efe9eb0c1c7b2]) successfully deleted
I20251023 20:15:31.554323 25068 ts_tablet_manager.cc:1916] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.553879 24780 pending_rounds.cc:77] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082156651958272 op_type: NO_OP noop_request { }
I20251023 20:15:31.555193 24780 raft_consensus.cc:2889] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:31.556210 24780 raft_consensus.cc:2272] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.563426 23786 tablet_bootstrap.cc:654] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.567191 23790 tablet_bootstrap.cc:492] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.566780 24780 ts_tablet_manager.cc:1916] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.567348 23790 ts_tablet_manager.cc:1403] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20251023 20:15:31.568171 23790 raft_consensus.cc:359] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.568470 23790 raft_consensus.cc:385] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.568573 23790 raft_consensus.cc:740] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.569188 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c58a406d564b048fb16a85aa0b754b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:31.569221 23790 consensus_queue.cc:260] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.569381 23528 raft_consensus.cc:3060] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.570693 23790 ts_tablet_manager.cc:1434] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251023 20:15:31.572322 23528 raft_consensus.cc:2468] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:31.574162 25040 raft_consensus.cc:493] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.573601 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:31.574323 25040 raft_consensus.cc:515] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.574934 25040 leader_election.cc:290] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.573696 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c58a406d564b048fb16a85aa0b754b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:31.575536 23790 tablet_bootstrap.cc:492] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.575642 23670 raft_consensus.cc:3060] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.575314 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:31.577452 23790 tablet_bootstrap.cc:654] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.577548 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:31.578728 23670 raft_consensus.cc:2468] T 19c58a406d564b048fb16a85aa0b754b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:31.579581 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:31.563437 25068 ts_tablet_manager.cc:1929] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.587728 25068 log.cc:1199] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/83dae34129794f49b43038d3b1636ea0
I20251023 20:15:31.588388 25068 ts_tablet_manager.cc:1950] T 83dae34129794f49b43038d3b1636ea0 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.587339 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:31.592231 23790 tablet_bootstrap.cc:492] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.592376 23790 ts_tablet_manager.cc:1403] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20251023 20:15:31.593343 23790 raft_consensus.cc:359] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.593514 23790 raft_consensus.cc:385] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.593562 23790 raft_consensus.cc:740] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.593783 23790 consensus_queue.cc:260] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.594434 23790 ts_tablet_manager.cc:1434] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:31.594735 23790 tablet_bootstrap.cc:492] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.596184 23786 tablet_bootstrap.cc:492] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.596477 23790 tablet_bootstrap.cc:654] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.596671 23786 ts_tablet_manager.cc:1403] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.000s
I20251023 20:15:31.597555 23786 raft_consensus.cc:359] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.597834 23786 raft_consensus.cc:385] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.597986 23786 raft_consensus.cc:740] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.598398 23786 consensus_queue.cc:260] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.599323 23786 ts_tablet_manager.cc:1434] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:31.599743 23786 tablet_bootstrap.cc:492] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.601574 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ae6a6e58274b8baf952eaaefde7127" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:31.601846 23397 raft_consensus.cc:3060] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.601855 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 83dae34129794f49b43038d3b1636ea0 (table default.Table_52af8fa4ff964f16b6fe2b43e94dddd4 [id=872a29a7d5234fc98b6dd83be48499cd]) successfully deleted
I20251023 20:15:31.604390 23397 raft_consensus.cc:2468] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:31.605118 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "419e41d7a37f4b888cb242b089c28790" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.605466 23397 raft_consensus.cc:2468] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:31.606426 23786 tablet_bootstrap.cc:654] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.606736 23790 tablet_bootstrap.cc:492] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.606882 23790 ts_tablet_manager.cc:1403] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
W20251023 20:15:31.607199 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e5720c96730425dafb550094cf936cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5720c96730425dafb550094cf936cd already in progress: creating tablet
I20251023 20:15:31.607653 23790 raft_consensus.cc:359] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.607812 23790 raft_consensus.cc:385] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.607863 23790 raft_consensus.cc:740] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:31.607904 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a9c5b69df4c04365b1ac1457fe8300cc because the tablet was not found. No further retry: Not found: Tablet not found: a9c5b69df4c04365b1ac1457fe8300cc
I20251023 20:15:31.608038 23790 consensus_queue.cc:260] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.608668 23790 ts_tablet_manager.cc:1434] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251023 20:15:31.608662 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1a64a7c2e1f242e1b54243489622bae5 because the tablet was not found. No further retry: Not found: Tablet not found: 1a64a7c2e1f242e1b54243489622bae5
I20251023 20:15:31.608083 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:31.608954 23790 tablet_bootstrap.cc:492] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.610718 23790 tablet_bootstrap.cc:654] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:31.611335 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d2ae19cc80194363b8a9c96fbafb4ebe because the tablet was not found. No further retry: Not found: Tablet not found: d2ae19cc80194363b8a9c96fbafb4ebe
I20251023 20:15:31.611680 25068 tablet_replica.cc:333] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.612864 25068 raft_consensus.cc:2243] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.613107 25068 raft_consensus.cc:2272] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.613716 25068 ts_tablet_manager.cc:1916] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:31.614085 23203 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92c50c4845524163834861438291a3ec consensus state has no leader
I20251023 20:15:31.615442 23215 catalog_manager.cc:5649] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:31.622215 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9047d641796b4f20b0de82b277ce877c) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:31.623133 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:31.626101 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.625403 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:31.627871 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.629143 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:31.629426 23670 raft_consensus.cc:2468] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:31.630224 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:31.630686 23668 raft_consensus.cc:2468] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:31.633353 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 0438593fbbdf47cebfb64121acad104a (table default.Table_e13d2545a0a54b91a8bcb64db45581cc [id=5d8522349b2648fda1e17a6fd1b301dd]) successfully deleted
I20251023 20:15:31.634862 23528 raft_consensus.cc:3060] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.631824 23215 catalog_manager.cc:5649] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:31.637200 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:31.642379 23786 tablet_bootstrap.cc:492] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.642593 23786 ts_tablet_manager.cc:1403] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.002s
I20251023 20:15:31.640019 23215 catalog_manager.cc:5649] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:31.643455 23786 raft_consensus.cc:359] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.643702 23786 raft_consensus.cc:385] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.644038 24780 ts_tablet_manager.cc:1929] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:31.643781 23786 raft_consensus.cc:740] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.643991 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:31.644676 24780 log.cc:1199] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/5346a221c78342359eb6e418bd12cd24
I20251023 20:15:31.644706 23786 consensus_queue.cc:260] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.646059 23787 tablet_bootstrap.cc:492] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.646258 23787 ts_tablet_manager.cc:1403] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20251023 20:15:31.647035 23787 raft_consensus.cc:359] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.647541 23786 ts_tablet_manager.cc:1434] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:31.647432 24780 ts_tablet_manager.cc:1950] T 5346a221c78342359eb6e418bd12cd24 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:31.645853 23215 catalog_manager.cc:5649] T b8e41ac34e84459581daef13914942b5 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:31.647939 23787 raft_consensus.cc:385] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.648005 23787 raft_consensus.cc:740] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.648233 23787 consensus_queue.cc:260] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.648352 23790 tablet_bootstrap.cc:492] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.648478 23790 ts_tablet_manager.cc:1403] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.002s
I20251023 20:15:31.649248 23790 raft_consensus.cc:359] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.649410 23790 raft_consensus.cc:385] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.649457 23790 raft_consensus.cc:740] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.649876 23790 consensus_queue.cc:260] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.650358 23790 ts_tablet_manager.cc:1434] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:31.650640 23790 tablet_bootstrap.cc:492] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.652354 23790 tablet_bootstrap.cc:654] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.649448 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 139) took 2889 ms. Trace:
I20251023 20:15:31.655344 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:31.655511 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.655592 25068 ts_tablet_manager.cc:1929] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.655689 25068 log.cc:1199] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/46f700ea3db04a4bad539c943e850a44
I20251023 20:15:31.655726 23670 raft_consensus.cc:2468] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:31.656144 23786 tablet_bootstrap.cc:492] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.655149 24780 rpcz_store.cc:276] 1023 20:15:28.760294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.760517 (+ 223us) service_pool.cc:225] Handling call
1023 20:15:31.545497 (+2784980us) ts_tablet_manager.cc:1151] Deleting tablet 5346a221c78342359eb6e418bd12cd24
1023 20:15:31.545813 (+ 316us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.611155 (+ 65342us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.643222 (+ 32067us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.649269 (+ 6047us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2784495}
I20251023 20:15:31.654320 23215 catalog_manager.cc:5649] T d1cd0a61827c49fd99dfd054c9a292f8 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:31.658475 24780 tablet_replica.cc:333] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:31.660895 23787 ts_tablet_manager.cc:1434] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251023 20:15:31.661044 25068 ts_tablet_manager.cc:1950] T 46f700ea3db04a4bad539c943e850a44 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.661713 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 46f700ea3db04a4bad539c943e850a44 (table default.Table_aba84ed1ff684cc690f31c5615ec1700 [id=0b966d7b9d4d4ad1bdf1c84d63c6be5a]) successfully deleted
I20251023 20:15:31.661271 24780 raft_consensus.cc:2243] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:31.662091 23786 tablet_bootstrap.cc:654] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.663897 24780 raft_consensus.cc:2272] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:31.665064 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e194d03a6aa946fbb2c79d91b196440b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e194d03a6aa946fbb2c79d91b196440b already in progress: creating tablet
I20251023 20:15:31.666047 23787 tablet_bootstrap.cc:492] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.664779 23215 catalog_manager.cc:5649] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } }
W20251023 20:15:31.668306 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 40c659c273c449939d0dd708d39b393f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40c659c273c449939d0dd708d39b393f already in progress: creating tablet
W20251023 20:15:31.668941 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1f9935f5e4f4daa9df8fe1611ed76c9 already in progress: creating tablet
W20251023 20:15:31.669535 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 39a62c97e0a3459e91c24c1c2f92389a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39a62c97e0a3459e91c24c1c2f92389a already in progress: creating tablet
W20251023 20:15:31.670049 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 370a790a053446b2b06314c7db242270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 370a790a053446b2b06314c7db242270 already in progress: creating tablet
I20251023 20:15:31.669598 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:31.670306 24780 ts_tablet_manager.cc:1916] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.672250 23787 tablet_bootstrap.cc:654] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.672313 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
W20251023 20:15:31.673632 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1822fdf1f5f44dd4a742aa3c2f8443cf already in progress: creating tablet
W20251023 20:15:31.674175 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
I20251023 20:15:31.674510 23471 leader_election.cc:304] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:31.674818 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e19859a65a144598e1612be89daa3a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e19859a65a144598e1612be89daa3a1 already in progress: creating tablet
I20251023 20:15:31.675056 24830 raft_consensus.cc:2804] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:31.675108 23790 tablet_bootstrap.cc:492] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.675251 23790 ts_tablet_manager.cc:1403] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
W20251023 20:15:31.675583 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 76380b341a304d6ba31648fbd1548f05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76380b341a304d6ba31648fbd1548f05 already in progress: creating tablet
I20251023 20:15:31.676214 23790 raft_consensus.cc:359] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.676394 23790 raft_consensus.cc:385] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:31.676386 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e36bb5e61d34b35a373bd6a51a7ac5e already in progress: creating tablet
I20251023 20:15:31.676455 23790 raft_consensus.cc:740] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:31.676654 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 453eea5cdd2e47f29577c61b40ee59c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453eea5cdd2e47f29577c61b40ee59c0 already in progress: creating tablet
I20251023 20:15:31.676661 23790 consensus_queue.cc:260] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.677348 23790 ts_tablet_manager.cc:1434] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251023 20:15:31.677510 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
I20251023 20:15:31.677726 23790 tablet_bootstrap.cc:492] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.677842 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:31.679966 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:31.681669 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:31.684933 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:31.685961 23790 tablet_bootstrap.cc:654] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.687007 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet f2a27706ecdb4ab78bb96c10a9f6757f (table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]). Expected version 2 got 0
I20251023 20:15:31.688362 23471 leader_election.cc:304] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:31.691431 23787 tablet_bootstrap.cc:492] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.691665 23787 ts_tablet_manager.cc:1403] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20251023 20:15:31.692713 23787 raft_consensus.cc:359] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.693480 23787 raft_consensus.cc:385] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.693830 23787 raft_consensus.cc:740] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.694337 23787 consensus_queue.cc:260] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.695684 23787 ts_tablet_manager.cc:1434] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:31.696872 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:31.697738 23787 tablet_bootstrap.cc:492] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.698309 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]). Expected version 1 got 0
I20251023 20:15:31.698670 25068 tablet_replica.cc:333] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.699654 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:31.700235 25068 raft_consensus.cc:2243] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.701763 23786 tablet_bootstrap.cc:492] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.701473 25068 raft_consensus.cc:2272] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.702174 23787 tablet_bootstrap.cc:654] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.702157 23786 ts_tablet_manager.cc:1403] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.005s
I20251023 20:15:31.702250 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:31.703038 23786 raft_consensus.cc:359] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.703290 23786 raft_consensus.cc:385] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.703378 23786 raft_consensus.cc:740] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.703644 23786 consensus_queue.cc:260] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:31.707789 23333 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:31.710412 23787 tablet_bootstrap.cc:492] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.712194 23787 ts_tablet_manager.cc:1403] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20251023 20:15:31.713564 23787 raft_consensus.cc:359] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.713758 23786 ts_tablet_manager.cc:1434] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251023 20:15:31.714422 23786 tablet_bootstrap.cc:492] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.716051 23787 raft_consensus.cc:385] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.716022 25068 ts_tablet_manager.cc:1916] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.716444 25144 raft_consensus.cc:2804] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.716460 23787 raft_consensus.cc:740] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.717693 23787 consensus_queue.cc:260] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.717666 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:31.719600 23787 ts_tablet_manager.cc:1434] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:31.722303 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:31.723498 25144 raft_consensus.cc:493] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.723706 25144 raft_consensus.cc:3060] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.724795 23787 tablet_bootstrap.cc:492] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.724934 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:31.726418 25144 raft_consensus.cc:515] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.726655 23787 tablet_bootstrap.cc:654] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.726668 23333 leader_election.cc:304] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:31.727437 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:31.729555 24822 raft_consensus.cc:2804] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:31.729684 24822 raft_consensus.cc:697] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:31.729139 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:31.729933 24822 consensus_queue.cc:237] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.733158 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:31.734092 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:31.734669 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:31.734927 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:31.736065 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:31.736506 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:31.716305 23786 tablet_bootstrap.cc:654] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.737540 25095 raft_consensus.cc:493] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.737699 25095 raft_consensus.cc:515] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.738376 25095 leader_election.cc:290] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.739159 24830 raft_consensus.cc:697] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:31.739519 24780 ts_tablet_manager.cc:1929] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:31.739768 24780 log.cc:1199] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/5964da70989b4e6cb77fe5d0a52ce9cf
W20251023 20:15:31.739372 23470 leader_election.cc:343] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 419e41d7a37f4b888cb242b089c28790
I20251023 20:15:31.740003 25144 leader_election.cc:290] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.740324 24780 ts_tablet_manager.cc:1950] T 5964da70989b4e6cb77fe5d0a52ce9cf P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:31.740926 23787 tablet_bootstrap.cc:492] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.741547 23787 ts_tablet_manager.cc:1403] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
W20251023 20:15:31.741822 23333 consensus_peers.cc:597] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8eaa2167f82475ca2f6a110aba288ec. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:31.740304 24830 consensus_queue.cc:237] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.742698 23787 raft_consensus.cc:359] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.742870 23787 raft_consensus.cc:385] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.748840 23787 raft_consensus.cc:740] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.749851 23787 consensus_queue.cc:260] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.751371 23787 ts_tablet_manager.cc:1434] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:31.751672 25061 raft_consensus.cc:493] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.755241 25061 raft_consensus.cc:515] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.756036 25061 leader_election.cc:290] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.755415 23787 tablet_bootstrap.cc:492] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.756863 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "377de4ea0bd6489284fac6b0fab52c8c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.758013 23605 leader_election.cc:343] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.758131 23787 tablet_bootstrap.cc:654] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.758776 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "377de4ea0bd6489284fac6b0fab52c8c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:31.759390 23604 leader_election.cc:343] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.759495 23604 leader_election.cc:304] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.759924 25061 raft_consensus.cc:2749] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.761906 23333 leader_election.cc:304] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:31.763233 23333 leader_election.cc:304] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:31.763484 24822 raft_consensus.cc:2804] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.764360 24822 raft_consensus.cc:493] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.764518 24822 raft_consensus.cc:3060] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.767169 24822 raft_consensus.cc:515] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.770987 23787 tablet_bootstrap.cc:492] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.771245 23787 ts_tablet_manager.cc:1403] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251023 20:15:31.772053 23787 raft_consensus.cc:359] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.773046 23787 raft_consensus.cc:385] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.773310 23787 raft_consensus.cc:740] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.773676 23787 consensus_queue.cc:260] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.775003 23787 ts_tablet_manager.cc:1434] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:31.775797 23787 tablet_bootstrap.cc:492] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.775873 25068 ts_tablet_manager.cc:1929] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:31.776129 25068 log.cc:1199] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/994a555b24e4432a84601c54d422aa5d
I20251023 20:15:31.777630 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:31.777844 23787 tablet_bootstrap.cc:654] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.778038 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:31.779403 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:31.779873 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:31.780458 25144 raft_consensus.cc:493] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.780616 25144 raft_consensus.cc:515] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.780735 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:31.781311 25144 leader_election.cc:290] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:31.782702 23470 leader_election.cc:343] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 25079532547d4175802b84877790662d
I20251023 20:15:31.786751 23786 tablet_bootstrap.cc:492] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.786968 23786 ts_tablet_manager.cc:1403] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.073s user 0.001s sys 0.015s
I20251023 20:15:31.787725 23786 raft_consensus.cc:359] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.788045 23786 raft_consensus.cc:385] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.788152 23786 raft_consensus.cc:740] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.788429 23786 consensus_queue.cc:260] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.788982 23786 ts_tablet_manager.cc:1434] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:31.791254 23786 tablet_bootstrap.cc:492] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.767488 25040 raft_consensus.cc:2804] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.792660 25040 raft_consensus.cc:493] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.792768 25040 raft_consensus.cc:3060] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.793067 23786 tablet_bootstrap.cc:654] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.795478 25040 raft_consensus.cc:515] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.796171 25040 leader_election.cc:290] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.797919 25140 raft_consensus.cc:493] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.798079 25140 raft_consensus.cc:515] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.799209 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d238e071cb4d49b173febd90bc909b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.799510 24830 raft_consensus.cc:493] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.769441 23333 leader_election.cc:304] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:31.799659 23604 leader_election.cc:343] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.799659 24830 raft_consensus.cc:515] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.783175 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 140) took 3022 ms. Trace:
I20251023 20:15:31.799801 24780 rpcz_store.cc:276] 1023 20:15:28.760900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.761103 (+ 203us) service_pool.cc:225] Handling call
1023 20:15:31.658070 (+2896967us) ts_tablet_manager.cc:1151] Deleting tablet 5964da70989b4e6cb77fe5d0a52ce9cf
1023 20:15:31.658253 (+ 183us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.716193 (+ 57940us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.739329 (+ 23136us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.783151 (+ 43822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896613}
I20251023 20:15:31.788369 23790 tablet_bootstrap.cc:492] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.800217 23790 ts_tablet_manager.cc:1403] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.123s user 0.004s sys 0.017s
I20251023 20:15:31.800401 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d238e071cb4d49b173febd90bc909b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.800891 23605 leader_election.cc:343] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.800688 25068 ts_tablet_manager.cc:1950] T 994a555b24e4432a84601c54d422aa5d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.800985 23605 leader_election.cc:304] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.800978 24830 leader_election.cc:290] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.801160 23790 raft_consensus.cc:359] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.801298 23790 raft_consensus.cc:385] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.801343 23790 raft_consensus.cc:740] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.801386 25061 raft_consensus.cc:2749] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.801513 23790 consensus_queue.cc:260] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:31.801875 23470 leader_election.cc:343] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: fb01b3f4ff7f41e98063ebf76abeace6
I20251023 20:15:31.802080 23790 ts_tablet_manager.cc:1434] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:31.802081 25140 leader_election.cc:290] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.802225 24780 tablet_replica.cc:333] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:31.802393 24780 raft_consensus.cc:2243] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:31.802681 24780 raft_consensus.cc:2272] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.803248 24780 ts_tablet_manager.cc:1916] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.804515 23787 tablet_bootstrap.cc:492] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.804703 23787 ts_tablet_manager.cc:1403] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20251023 20:15:31.805260 25040 raft_consensus.cc:2804] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.805347 25040 raft_consensus.cc:493] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.805418 25040 raft_consensus.cc:3060] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.805637 23787 raft_consensus.cc:359] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.806396 23787 raft_consensus.cc:385] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.806505 23787 raft_consensus.cc:740] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.806758 23787 consensus_queue.cc:260] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.807919 25040 raft_consensus.cc:515] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.808576 25040 leader_election.cc:290] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.809399 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]) successfully deleted
W20251023 20:15:31.809726 23333 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:31.809834 23333 leader_election.cc:304] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.810073 24821 raft_consensus.cc:2749] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:31.810511 23333 consensus_peers.cc:597] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8eaa2167f82475ca2f6a110aba288ec. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:31.810529 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
W20251023 20:15:31.811702 23333 leader_election.cc:343] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:31.811983 23333 leader_election.cc:343] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:31.812615 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ca11ddd5d97544cf916b87fd3dff219d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca11ddd5d97544cf916b87fd3dff219d already in progress: creating tablet
I20251023 20:15:31.813369 23790 tablet_bootstrap.cc:492] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:31.814911 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f9bd2b109bfa4804a372965e60ba6e85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bd2b109bfa4804a372965e60ba6e85 already in progress: creating tablet
I20251023 20:15:31.815032 23790 tablet_bootstrap.cc:654] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.815415 23787 ts_tablet_manager.cc:1434] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251023 20:15:31.815793 23787 tablet_bootstrap.cc:492] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.816076 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
W20251023 20:15:31.817510 23333 leader_election.cc:343] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.817950 23786 tablet_bootstrap.cc:492] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.818120 23786 ts_tablet_manager.cc:1403] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
W20251023 20:15:31.818333 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 8317a54a5f7b4afe8da9d336268efe43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8317a54a5f7b4afe8da9d336268efe43 already in progress: creating tablet
I20251023 20:15:31.818867 23786 raft_consensus.cc:359] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.819036 23786 raft_consensus.cc:385] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.819097 23786 raft_consensus.cc:740] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.819285 23786 consensus_queue.cc:260] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.820367 25144 raft_consensus.cc:493] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.820489 25144 raft_consensus.cc:515] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.820652 23787 tablet_bootstrap.cc:654] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.821053 25144 leader_election.cc:290] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.821709 23786 ts_tablet_manager.cc:1434] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:31.822268 23786 tablet_bootstrap.cc:492] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:31.822423 23470 leader_election.cc:343] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 05933cb9433d48d285b2db82b289bdc8
I20251023 20:15:31.823396 25140 raft_consensus.cc:493] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.823535 25140 raft_consensus.cc:515] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.824045 23786 tablet_bootstrap.cc:654] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.824050 24822 leader_election.cc:290] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.824182 25140 leader_election.cc:290] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.824784 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b766ff2dff34a8f80f70591dba54017" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.824985 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_d22911da756748fb8939006df660e885 [id=78fe983663c94eabab93ee902bde6998]). Expected version 5 got 4
W20251023 20:15:31.825482 23604 leader_election.cc:343] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.825699 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
W20251023 20:15:31.825960 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a4f012469fa4e8e827e485287ad76dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a4f012469fa4e8e827e485287ad76dd already in progress: creating tablet
I20251023 20:15:31.826931 24830 raft_consensus.cc:493] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.827049 24830 raft_consensus.cc:515] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.827597 24830 leader_election.cc:290] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:31.828604 23333 consensus_peers.cc:597] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbacaf2fea24405caaf84d59b625fca3. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:31.829288 23470 leader_election.cc:343] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: c9bfbf0acdc84f2f8fe553694ee9897c
W20251023 20:15:31.829599 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d951134f6773410386bc57f7f90310c3 because the tablet was not found. No further retry: Not found: Tablet not found: d951134f6773410386bc57f7f90310c3
I20251023 20:15:31.828614 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b766ff2dff34a8f80f70591dba54017" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.830834 23605 leader_election.cc:343] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.830955 23605 leader_election.cc:304] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.831130 25092 consensus_queue.cc:1048] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:31.831470 25140 raft_consensus.cc:2749] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.832109 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 5346a221c78342359eb6e418bd12cd24 (table default.Table_54bd013a97df4bb08fdab2eeea9f5d42 [id=71ba16e8b75642e09686eb46c4315366]) successfully deleted
W20251023 20:15:31.833474 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d62dbf044acd4459b26d2e7e50f6206e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d62dbf044acd4459b26d2e7e50f6206e already in progress: creating tablet
I20251023 20:15:31.834182 25068 tablet_replica.cc:333] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.834901 25068 raft_consensus.cc:2243] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.835496 25068 raft_consensus.cc:2272] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.839622 25068 ts_tablet_manager.cc:1916] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.842175 23786 tablet_bootstrap.cc:492] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.842376 23786 ts_tablet_manager.cc:1403] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20251023 20:15:31.842665 23787 tablet_bootstrap.cc:492] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.842846 23787 ts_tablet_manager.cc:1403] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.005s
I20251023 20:15:31.843173 23786 raft_consensus.cc:359] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.843336 23786 raft_consensus.cc:385] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.843387 23786 raft_consensus.cc:740] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.843605 23786 consensus_queue.cc:260] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.843789 23787 raft_consensus.cc:359] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.843984 23787 raft_consensus.cc:385] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.844071 23787 raft_consensus.cc:740] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.825986 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:31.844261 23786 ts_tablet_manager.cc:1434] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.844398 23787 consensus_queue.cc:260] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.845168 23787 ts_tablet_manager.cc:1434] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:31.845463 23787 tablet_bootstrap.cc:492] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.845057 23215 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:31.847127 23787 tablet_bootstrap.cc:654] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.848637 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:31.853662 25140 raft_consensus.cc:493] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.853792 25140 raft_consensus.cc:515] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.854339 25140 leader_election.cc:290] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.854633 23786 tablet_bootstrap.cc:492] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.855134 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:31.855721 23604 leader_election.cc:343] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.856331 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb475e303d07491bbc19dc3f628bceb7" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.856338 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.856825 23605 leader_election.cc:343] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.856947 23605 leader_election.cc:304] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.857406 25140 raft_consensus.cc:2749] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.860257 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:31.860538 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:31.860698 23668 raft_consensus.cc:3060] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.863567 23668 raft_consensus.cc:2468] T 5407a850ec7045158f53cb8abf6fcb58 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:31.866000 23786 tablet_bootstrap.cc:654] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.866459 23787 tablet_bootstrap.cc:492] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.866685 23787 ts_tablet_manager.cc:1403] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20251023 20:15:31.867437 23787 raft_consensus.cc:359] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.869973 23790 tablet_bootstrap.cc:492] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.870339 25140 raft_consensus.cc:493] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.870456 25140 raft_consensus.cc:515] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.870643 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 5964da70989b4e6cb77fe5d0a52ce9cf (table default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]) successfully deleted
I20251023 20:15:31.871004 25140 leader_election.cc:290] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.871850 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94cd8880a78c4b4eb0c19ba5944b12f2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.872225 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94cd8880a78c4b4eb0c19ba5944b12f2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.872562 23790 ts_tablet_manager.cc:1403] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
W20251023 20:15:31.872735 23604 leader_election.cc:343] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:31.873040 23605 leader_election.cc:343] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.873136 23605 leader_election.cc:304] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.873669 25140 raft_consensus.cc:2749] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.875798 24830 raft_consensus.cc:493] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.875924 24830 raft_consensus.cc:515] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.876452 24830 leader_election.cc:290] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.876709 23218 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 73) took 4951 ms. Trace:
I20251023 20:15:31.876825 23218 rpcz_store.cc:276] 1023 20:15:26.925074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.115718 (+4190644us) service_pool.cc:225] Handling call
1023 20:15:31.205095 (+ 89377us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a060f4eb 00007f8997113727 00007f8997113455 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:31.876644 (+671549us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204778752}
I20251023 20:15:31.876925 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:31.877159 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:31.877275 23218 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_c709570632244bd3af291b4e143d2eb5" } modify_external_catalogs: true
I20251023 20:15:31.877453 23218 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_c709570632244bd3af291b4e143d2eb5" } modify_external_catalogs: true
I20251023 20:15:31.877593 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f7dffd94f3442ea20045a512d7bd65" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:31.877737 23670 raft_consensus.cc:2468] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:31.878489 23470 leader_election.cc:304] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:31.878886 24830 raft_consensus.cc:2804] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.878957 24830 raft_consensus.cc:493] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.879009 24830 raft_consensus.cc:3060] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.881196 23668 raft_consensus.cc:3060] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.880349 23790 raft_consensus.cc:359] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.882589 25144 raft_consensus.cc:493] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.882714 25144 raft_consensus.cc:515] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.882251 23790 raft_consensus.cc:385] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.883314 25144 leader_election.cc:290] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.882977 23790 raft_consensus.cc:740] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.884079 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
W20251023 20:15:31.884408 23470 leader_election.cc:343] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a86a25659ec44d7c825783036342ed45
I20251023 20:15:31.885653 23790 consensus_queue.cc:260] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.892364 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:31.889609 24830 raft_consensus.cc:515] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.894215 23786 tablet_bootstrap.cc:492] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.894402 23786 ts_tablet_manager.cc:1403] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.005s
I20251023 20:15:31.894591 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f7dffd94f3442ea20045a512d7bd65" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:31.894796 23670 raft_consensus.cc:3060] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.895206 23786 raft_consensus.cc:359] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.895365 23786 raft_consensus.cc:385] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.895411 23786 raft_consensus.cc:740] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.895627 23786 consensus_queue.cc:260] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.896234 23786 ts_tablet_manager.cc:1434] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:31.896481 23786 tablet_bootstrap.cc:492] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.897795 24830 leader_election.cc:290] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.898239 23786 tablet_bootstrap.cc:654] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.899235 23670 raft_consensus.cc:2468] T 04f7dffd94f3442ea20045a512d7bd65 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:31.902243 23790 ts_tablet_manager.cc:1434] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20251023 20:15:31.900374 23470 leader_election.cc:304] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:31.909310 23668 raft_consensus.cc:2468] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:31.909417 24830 raft_consensus.cc:2804] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:31.909644 24830 raft_consensus.cc:697] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:31.909953 24830 consensus_queue.cc:237] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.913884 23790 tablet_bootstrap.cc:492] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.914811 23787 raft_consensus.cc:385] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.914992 23787 raft_consensus.cc:740] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.915292 23787 consensus_queue.cc:260] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.924535 23787 ts_tablet_manager.cc:1434] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.058s user 0.016s sys 0.000s
I20251023 20:15:31.925549 25068 ts_tablet_manager.cc:1929] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:31.926353 25061 raft_consensus.cc:493] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.926476 25061 raft_consensus.cc:515] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.926806 25068 log.cc:1199] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/419e41d7a37f4b888cb242b089c28790
I20251023 20:15:31.927057 25061 leader_election.cc:290] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.927376 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72d1dde8b0ac4812869ef233c18e550f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.927598 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72d1dde8b0ac4812869ef233c18e550f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:31.927948 23604 leader_election.cc:343] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:31.928119 23605 leader_election.cc:343] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.928236 23605 leader_election.cc:304] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:31.928629 25061 raft_consensus.cc:2749] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.928779 23790 tablet_bootstrap.cc:654] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.929013 25144 raft_consensus.cc:493] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.929144 25144 raft_consensus.cc:515] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.929804 25144 leader_election.cc:290] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.930394 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:31.931022 25068 ts_tablet_manager.cc:1950] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:31.931408 23787 tablet_bootstrap.cc:492] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:31.932785 23787 tablet_bootstrap.cc:654] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.935310 24780 ts_tablet_manager.cc:1929] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:31.935423 24780 log.cc:1199] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/9fc7e18054c440098dc2470143ea7fae
I20251023 20:15:31.935750 24780 ts_tablet_manager.cc:1950] T 9fc7e18054c440098dc2470143ea7fae P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:31.935117 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 348) took 1078 ms. Trace:
I20251023 20:15:31.936491 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 141) took 3175 ms. Trace:
I20251023 20:15:31.936586 24780 rpcz_store.cc:276] 1023 20:15:28.761450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.761606 (+ 156us) service_pool.cc:225] Handling call
1023 20:15:31.800100 (+3038494us) ts_tablet_manager.cc:1151] Deleting tablet 9fc7e18054c440098dc2470143ea7fae
1023 20:15:31.802174 (+ 2074us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.836336 (+ 34162us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.935176 (+ 98840us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.936467 (+ 1291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3038349}
I20251023 20:15:31.936882 24780 tablet_replica.cc:333] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:31.935998 25068 rpcz_store.cc:276] 1023 20:15:30.856327 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.857216 (+ 889us) service_pool.cc:225] Handling call
1023 20:15:31.833938 (+976722us) ts_tablet_manager.cc:1151] Deleting tablet 419e41d7a37f4b888cb242b089c28790
1023 20:15:31.834039 (+ 101us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.879111 (+ 45072us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.924546 (+ 45435us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:31.934869 (+ 10323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":976537}
I20251023 20:15:31.938153 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 419e41d7a37f4b888cb242b089c28790 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:31.938872 25144 raft_consensus.cc:493] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.939020 25144 raft_consensus.cc:515] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.939668 25144 leader_election.cc:290] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.940140 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 350) took 1083 ms. Trace:
I20251023 20:15:31.941211 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:31.940445 25068 rpcz_store.cc:276] 1023 20:15:30.856504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.856859 (+ 355us) service_pool.cc:225] Handling call
1023 20:15:31.938733 (+1081874us) ts_tablet_manager.cc:1151] Deleting tablet 01fa5f9c700b458ea147373d78551573
1023 20:15:31.938944 (+ 211us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.939965 (+ 1021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081317}
W20251023 20:15:31.941737 23470 leader_election.cc:343] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 9ed0aa7e36e64843ad6b2ad4b415832a
W20251023 20:15:31.942072 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 01fa5f9c700b458ea147373d78551573 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01fa5f9c700b458ea147373d78551573 already in progress: creating tablet
I20251023 20:15:31.942090 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:31.942255 23668 raft_consensus.cc:3060] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.943773 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "419e41d7a37f4b888cb242b089c28790" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:31.943949 23397 raft_consensus.cc:3060] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.944749 23668 raft_consensus.cc:2468] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:31.945119 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 347) took 1088 ms. Trace:
I20251023 20:15:31.946542 23397 raft_consensus.cc:2468] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:31.945729 25068 rpcz_store.cc:276] 1023 20:15:30.856217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.857424 (+ 1207us) service_pool.cc:225] Handling call
1023 20:15:31.942892 (+1085468us) ts_tablet_manager.cc:1151] Deleting tablet 377de4ea0bd6489284fac6b0fab52c8c
1023 20:15:31.943125 (+ 233us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.944904 (+ 1779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085313}
W20251023 20:15:31.947552 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 377de4ea0bd6489284fac6b0fab52c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377de4ea0bd6489284fac6b0fab52c8c already in progress: creating tablet
I20251023 20:15:31.947645 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25079532547d4175802b84877790662d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.947808 23397 raft_consensus.cc:2468] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:31.948184 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb01b3f4ff7f41e98063ebf76abeace6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.948331 23397 raft_consensus.cc:2468] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:31.948446 23786 tablet_bootstrap.cc:492] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:31.948712 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "05933cb9433d48d285b2db82b289bdc8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.949184 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bfbf0acdc84f2f8fe553694ee9897c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.949463 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 346) took 1093 ms. Trace:
I20251023 20:15:31.949420 23786 ts_tablet_manager.cc:1403] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20251023 20:15:31.950219 25068 rpcz_store.cc:276] 1023 20:15:30.856132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.857591 (+ 1459us) service_pool.cc:225] Handling call
1023 20:15:31.948251 (+1090660us) ts_tablet_manager.cc:1151] Deleting tablet 0858358aabff47a69dc77f4669b225c5
1023 20:15:31.948585 (+ 334us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.949202 (+ 617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090529}
W20251023 20:15:31.951937 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 0858358aabff47a69dc77f4669b225c5 because the tablet was not found. No further retry: Not found: Tablet not found: 0858358aabff47a69dc77f4669b225c5
I20251023 20:15:31.954453 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 345) took 1098 ms. Trace:
I20251023 20:15:31.955152 25068 rpcz_store.cc:276] 1023 20:15:30.856044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.857738 (+ 1694us) service_pool.cc:225] Handling call
1023 20:15:31.953280 (+1095542us) ts_tablet_manager.cc:1151] Deleting tablet a8ba7ab28dfa4ac38e0b58ba2115495c
1023 20:15:31.953292 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.954211 (+ 919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1094673}
I20251023 20:15:31.956899 23790 tablet_bootstrap.cc:492] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:31.956902 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a8ba7ab28dfa4ac38e0b58ba2115495c because the tablet was not found. No further retry: Not found: Tablet not found: a8ba7ab28dfa4ac38e0b58ba2115495c
I20251023 20:15:31.957127 23790 ts_tablet_manager.cc:1403] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20251023 20:15:31.957895 23790 raft_consensus.cc:359] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.958065 23790 raft_consensus.cc:385] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.958114 23790 raft_consensus.cc:740] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.958426 23790 consensus_queue.cc:260] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.958909 23790 ts_tablet_manager.cc:1434] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:31.959213 23790 tablet_bootstrap.cc:492] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:31.958259 23786 raft_consensus.cc:359] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.959370 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 329) took 1104 ms. Trace:
I20251023 20:15:31.959749 23786 raft_consensus.cc:385] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.960368 23786 raft_consensus.cc:740] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:31.959992 25068 rpcz_store.cc:276] 1023 20:15:30.854408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.857936 (+ 3528us) service_pool.cc:225] Handling call
1023 20:15:31.957993 (+1100057us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:31.958010 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.959134 (+ 1124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099684}
I20251023 20:15:31.961251 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f7dffd94f3442ea20045a512d7bd65" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.961421 23397 raft_consensus.cc:2468] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
W20251023 20:15:31.961658 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 already in progress: creating tablet
I20251023 20:15:31.961881 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a86a25659ec44d7c825783036342ed45" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.962029 23397 raft_consensus.cc:2468] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:31.962393 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f7dffd94f3442ea20045a512d7bd65" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:31.962522 23397 raft_consensus.cc:3060] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.963105 25068 tablet_replica.cc:333] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:31.962793 23786 consensus_queue.cc:260] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.964190 23790 tablet_bootstrap.cc:654] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.964708 25068 raft_consensus.cc:2243] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:31.965620 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ec606cbf834ccd8762e034b520e58e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.966120 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed0aa7e36e64843ad6b2ad4b415832a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:31.966269 23395 raft_consensus.cc:2468] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:31.966563 25068 raft_consensus.cc:2272] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:31.966971 23534 raft_consensus.cc:3060] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.968691 23397 raft_consensus.cc:2468] T 04f7dffd94f3442ea20045a512d7bd65 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:31.968767 23786 ts_tablet_manager.cc:1434] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20251023 20:15:31.971235 23786 tablet_bootstrap.cc:492] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:31.970804 25068 ts_tablet_manager.cc:1916] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:31.976533 23790 tablet_bootstrap.cc:492] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:31.976823 25061 raft_consensus.cc:493] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.976959 25061 raft_consensus.cc:515] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.977543 25061 leader_election.cc:290] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.978514 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29c2c976d6a3440394f478a31abc7134" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:31.979101 23604 leader_election.cc:343] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.979295 25061 raft_consensus.cc:493] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.979416 25061 raft_consensus.cc:515] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.979954 25061 leader_election.cc:290] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.980684 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:31.980846 23790 ts_tablet_manager.cc:1403] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
W20251023 20:15:31.981326 23604 leader_election.cc:343] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:31.981665 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 9fc7e18054c440098dc2470143ea7fae (table default.Table_265f006bfcc2450f964410c799f035f9 [id=f0872a169a604701b06c246f1af67ac0]) successfully deleted
I20251023 20:15:31.984165 23471 leader_election.cc:304] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:31.984347 23471 raft_consensus.cc:2686] T 419e41d7a37f4b888cb242b089c28790 P ea6853efbbcc484eb7529c7e9b37a47a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251023 20:15:31.984647 24821 raft_consensus.cc:493] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.984966 23471 leader_election.cc:304] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:31.985255 24822 consensus_queue.cc:1048] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:31.985477 23471 leader_election.cc:304] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:31.985728 25139 raft_consensus.cc:2804] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.985805 25139 raft_consensus.cc:493] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.985877 25139 raft_consensus.cc:3060] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.985849 23471 leader_election.cc:304] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:31.986032 25144 raft_consensus.cc:2804] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.986091 25144 raft_consensus.cc:493] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.986140 25144 raft_consensus.cc:3060] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.988422 25139 raft_consensus.cc:515] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.988477 25144 raft_consensus.cc:515] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.989058 25139 leader_election.cc:290] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.989504 23471 leader_election.cc:304] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:31.991132 23471 leader_election.cc:304] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:31.990168 23470 leader_election.cc:343] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: fb01b3f4ff7f41e98063ebf76abeace6
W20251023 20:15:31.991438 23470 leader_election.cc:343] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 25079532547d4175802b84877790662d
I20251023 20:15:31.991824 25142 raft_consensus.cc:2749] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:31.989195 25144 leader_election.cc:290] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.991983 25142 raft_consensus.cc:2804] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:31.992092 25142 raft_consensus.cc:493] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:31.992230 25142 raft_consensus.cc:3060] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:31.992610 23786 tablet_bootstrap.cc:654] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:31.989362 24830 raft_consensus.cc:2749] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:31.994887 25142 raft_consensus.cc:515] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.994367 23787 tablet_bootstrap.cc:492] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:31.995157 23787 ts_tablet_manager.cc:1403] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.003s
I20251023 20:15:31.995450 25142 leader_election.cc:290] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:31.995894 23787 raft_consensus.cc:359] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.996596 23787 raft_consensus.cc:385] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:31.996703 23787 raft_consensus.cc:740] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:31.996768 23470 leader_election.cc:343] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: a86a25659ec44d7c825783036342ed45
I20251023 20:15:31.992092 25144 consensus_queue.cc:1048] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251023 20:15:31.997133 23787 consensus_queue.cc:260] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.997838 23787 ts_tablet_manager.cc:1434] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:31.998088 25061 raft_consensus.cc:493] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:31.998185 25061 raft_consensus.cc:515] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:31.998728 25061 leader_election.cc:290] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:31.999758 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.000294 23604 leader_election.cc:343] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.000694 23787 tablet_bootstrap.cc:492] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.001389 25142 raft_consensus.cc:493] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.001605 25142 raft_consensus.cc:515] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.002255 25142 leader_election.cc:290] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.002766 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.002943 23670 raft_consensus.cc:2468] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.003420 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.003499 23470 leader_election.cc:304] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.003798 25142 raft_consensus.cc:2804] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:15:32.003903 25142 raft_consensus.cc:493] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.004066 25142 raft_consensus.cc:3060] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Advancing to term 2
W20251023 20:15:32.004070 23605 leader_election.cc:343] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.004184 23605 leader_election.cc:304] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.004392 25061 raft_consensus.cc:2749] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.004313 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:32.005501 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:32.006524 23790 raft_consensus.cc:359] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.006785 24821 raft_consensus.cc:515] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.006880 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:32.007340 24821 leader_election.cc:290] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.007792 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29c2c976d6a3440394f478a31abc7134" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:32.008308 23605 leader_election.cc:343] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.008415 23605 leader_election.cc:304] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.008788 25061 raft_consensus.cc:2749] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.008852 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:32.009317 23605 leader_election.cc:343] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:32.009370 23333 leader_election.cc:343] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: bb475e303d07491bbc19dc3f628bceb7
I20251023 20:15:32.009413 23605 leader_election.cc:304] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.009981 25061 raft_consensus.cc:2749] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:32.010251 23333 consensus_peers.cc:597] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9bfbf0acdc84f2f8fe553694ee9897c. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:32.010365 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:32.010757 23787 tablet_bootstrap.cc:654] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.011232 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:32.012446 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:32.013548 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:32.014158 23471 leader_election.cc:304] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:32.014688 25142 raft_consensus.cc:515] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.014736 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:32.015102 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:32.015390 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:32.015664 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:32.016041 23471 leader_election.cc:304] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:32.017300 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.017473 23670 raft_consensus.cc:3060] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:32.019937 23670 raft_consensus.cc:2468] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 2.
I20251023 20:15:32.015937 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:32.020608 25144 raft_consensus.cc:2749] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.020835 25144 raft_consensus.cc:2804] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.021145 25144 raft_consensus.cc:493] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.021157 24780 raft_consensus.cc:2243] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:32.021191 23790 raft_consensus.cc:385] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.021263 25144 raft_consensus.cc:3060] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.021292 23790 raft_consensus.cc:740] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.021543 23790 consensus_queue.cc:260] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.021693 24780 raft_consensus.cc:2272] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.021600 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:32.022109 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:32.022167 23790 ts_tablet_manager.cc:1434] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20251023 20:15:32.022379 24780 ts_tablet_manager.cc:1916] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.022462 23470 leader_election.cc:304] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.022707 25142 leader_election.cc:290] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.024636 24830 raft_consensus.cc:493] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.025592 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:32.025965 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:32.026288 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:32.026364 25068 ts_tablet_manager.cc:1929] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:32.026607 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:32.026932 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:32.027405 25068 log.cc:1199] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/d9f73477a06542c0b215ae86716cdc4e
I20251023 20:15:32.027446 25144 raft_consensus.cc:515] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.028079 25068 ts_tablet_manager.cc:1950] T d9f73477a06542c0b215ae86716cdc4e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:32.028347 25144 leader_election.cc:290] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.027467 24830 raft_consensus.cc:515] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:32.029361 23470 leader_election.cc:343] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 9ed0aa7e36e64843ad6b2ad4b415832a
I20251023 20:15:32.029399 23534 raft_consensus.cc:3060] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.029373 24830 leader_election.cc:290] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.027555 25142 raft_consensus.cc:2804] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:15:32.029876 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e4189623c7a4ca1bdf73f71663e2b77" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.029956 25142 raft_consensus.cc:697] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:32.030115 23670 raft_consensus.cc:2468] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:32.030509 23470 leader_election.cc:304] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.030381 25142 consensus_queue.cc:237] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.030828 24830 raft_consensus.cc:2804] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.030917 24830 raft_consensus.cc:493] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.031019 24830 raft_consensus.cc:3060] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.032577 23333 leader_election.cc:304] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.032902 23333 leader_election.cc:304] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.033141 25040 raft_consensus.cc:2804] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
W20251023 20:15:32.033216 23333 consensus_peers.cc:597] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05933cb9433d48d285b2db82b289bdc8. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:32.033298 25040 raft_consensus.cc:697] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.033378 23333 leader_election.cc:304] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.033541 24830 raft_consensus.cc:515] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.033563 25040 consensus_queue.cc:237] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.033979 24821 raft_consensus.cc:2804] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.034085 24830 leader_election.cc:290] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.034535 24297 raft_consensus.cc:2804] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.034658 24297 raft_consensus.cc:697] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.034893 24297 consensus_queue.cc:237] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.035269 23790 tablet_bootstrap.cc:492] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.035807 25095 consensus_queue.cc:1048] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.036984 23790 tablet_bootstrap.cc:654] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.042110 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e4189623c7a4ca1bdf73f71663e2b77" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.042335 23670 raft_consensus.cc:3060] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.051122 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:32.045056 23670 raft_consensus.cc:2468] T 6e4189623c7a4ca1bdf73f71663e2b77 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.053112 23786 tablet_bootstrap.cc:492] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.053287 23470 leader_election.cc:304] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.053658 25167 raft_consensus.cc:2804] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.053830 25167 raft_consensus.cc:697] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:32.054138 25167 consensus_queue.cc:237] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.053289 23786 ts_tablet_manager.cc:1403] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.001s
I20251023 20:15:32.060753 23786 raft_consensus.cc:359] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.060992 23786 raft_consensus.cc:385] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.061441 23786 raft_consensus.cc:740] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.061759 23786 consensus_queue.cc:260] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.062714 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 351) took 1176 ms. Trace:
I20251023 20:15:32.062958 25068 rpcz_store.cc:276] 1023 20:15:30.856593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.858116 (+ 1523us) service_pool.cc:225] Handling call
1023 20:15:31.962826 (+1104710us) ts_tablet_manager.cc:1151] Deleting tablet d9f73477a06542c0b215ae86716cdc4e
1023 20:15:31.962839 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:31.999133 (+ 36294us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.013606 (+ 14473us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.032979 (+ 19373us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2934,"tablet-delete.queue_time_us":1104222}
I20251023 20:15:32.063813 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 349) took 1207 ms. Trace:
I20251023 20:15:32.064029 25068 rpcz_store.cc:276] 1023 20:15:30.856414 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.858270 (+ 1856us) service_pool.cc:225] Handling call
1023 20:15:32.063512 (+1205242us) ts_tablet_manager.cc:1151] Deleting tablet 951b2276d3324c2d999c8d7fb97767bc
1023 20:15:32.063525 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.063703 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205080}
I20251023 20:15:32.064677 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 330) took 1210 ms. Trace:
I20251023 20:15:32.064908 25068 rpcz_store.cc:276] 1023 20:15:30.854529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.858981 (+ 4452us) service_pool.cc:225] Handling call
1023 20:15:32.064451 (+1205470us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:32.064463 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.064644 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205295}
I20251023 20:15:32.065508 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 353) took 1157 ms. Trace:
I20251023 20:15:32.065722 25068 rpcz_store.cc:276] 1023 20:15:30.908288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.908361 (+ 73us) service_pool.cc:225] Handling call
1023 20:15:32.065305 (+1156944us) ts_tablet_manager.cc:1151] Deleting tablet 1ae910ac41fd4e349f159ab27e44375a
1023 20:15:32.065316 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.065450 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1156553}
I20251023 20:15:32.066264 25068 tablet_replica.cc:333] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:32.066608 25068 raft_consensus.cc:2243] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:32.067174 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet d9f73477a06542c0b215ae86716cdc4e (table default.Table_79be15cb4350452b90277004ec803c8a [id=6604e04478d9438e990e19240dd10e73]) successfully deleted
W20251023 20:15:32.067551 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 951b2276d3324c2d999c8d7fb97767bc because the tablet was not found. No further retry: Not found: Tablet not found: 951b2276d3324c2d999c8d7fb97767bc
W20251023 20:15:32.067783 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 857da3642f0846ce92aaed0491e89644 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 857da3642f0846ce92aaed0491e89644 already in progress: creating tablet
W20251023 20:15:32.067991 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1ae910ac41fd4e349f159ab27e44375a because the tablet was not found. No further retry: Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:32.068825 24780 ts_tablet_manager.cc:1929] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.068984 24780 log.cc:1199] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/7645c36c983f438f8b1223256fd3ad2c
I20251023 20:15:32.069386 24780 ts_tablet_manager.cc:1950] T 7645c36c983f438f8b1223256fd3ad2c P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.069764 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 142) took 3307 ms. Trace:
I20251023 20:15:32.069862 24780 rpcz_store.cc:276] 1023 20:15:28.761926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.762084 (+ 158us) service_pool.cc:225] Handling call
1023 20:15:31.936836 (+3174752us) ts_tablet_manager.cc:1151] Deleting tablet 7645c36c983f438f8b1223256fd3ad2c
1023 20:15:31.936845 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.026780 (+ 89935us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.068663 (+ 41883us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.069711 (+ 1048us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33,"spinlock_wait_cycles":69376,"tablet-delete.queue_time_us":3174591}
I20251023 20:15:32.070250 24780 tablet_replica.cc:333] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:32.070454 24780 raft_consensus.cc:2243] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.070676 24780 raft_consensus.cc:2272] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.071300 24780 ts_tablet_manager.cc:1916] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.071849 23787 tablet_bootstrap.cc:492] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.072052 23787 ts_tablet_manager.cc:1403] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.071s user 0.009s sys 0.015s
I20251023 20:15:32.072763 23787 raft_consensus.cc:359] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.073021 23787 raft_consensus.cc:385] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.073128 23787 raft_consensus.cc:740] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.067171 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:32.073403 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:32.073457 23787 consensus_queue.cc:260] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.074154 23787 ts_tablet_manager.cc:1434] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:32.074462 23787 tablet_bootstrap.cc:492] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.075995 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:32.076306 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:32.076864 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:32.077378 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:32.077853 25061 raft_consensus.cc:493] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.077956 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:32.077986 25061 raft_consensus.cc:515] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.078570 25061 leader_election.cc:290] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.079654 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6480cdc1afda4671a346044f232b083e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.080263 23604 leader_election.cc:343] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.082367 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6480cdc1afda4671a346044f232b083e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:32.083163 23605 leader_election.cc:343] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.083282 23605 leader_election.cc:304] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:32.083784 23470 consensus_peers.cc:597] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac55e14745534a02a2f6a422cab6b149. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:32.084029 25061 raft_consensus.cc:2749] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.084851 25068 pending_rounds.cc:70] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Trying to abort 1 pending ops.
I20251023 20:15:32.085038 25068 pending_rounds.cc:77] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082178103762944 op_type: NO_OP noop_request { }
I20251023 20:15:32.085222 25068 raft_consensus.cc:2889] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:32.085419 25068 raft_consensus.cc:2272] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.062834 23786 ts_tablet_manager.cc:1434] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:32.087685 23786 tablet_bootstrap.cc:492] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.078240 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:32.087620 25068 ts_tablet_manager.cc:1916] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.088402 24821 raft_consensus.cc:697] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.088893 24821 consensus_queue.cc:237] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.089200 23787 tablet_bootstrap.cc:654] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.089563 23786 tablet_bootstrap.cc:654] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.089983 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:32.091297 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:32.091734 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:32.092036 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:32.092458 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:32.098636 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:32.099320 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:32.099560 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:32.099761 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:32.100340 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:32.101354 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:32.102634 23787 tablet_bootstrap.cc:492] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.105340 23787 ts_tablet_manager.cc:1403] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20251023 20:15:32.105315 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_d22911da756748fb8939006df660e885 [id=78fe983663c94eabab93ee902bde6998]). Expected version 5 got 4
I20251023 20:15:32.105713 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:32.106230 23787 raft_consensus.cc:359] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.106395 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:32.106489 23787 raft_consensus.cc:385] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.106576 23787 raft_consensus.cc:740] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.106846 23787 consensus_queue.cc:260] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.107362 23786 tablet_bootstrap.cc:492] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.107589 23786 ts_tablet_manager.cc:1403] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20251023 20:15:32.108400 23786 raft_consensus.cc:359] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.108666 23786 raft_consensus.cc:385] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.108788 23786 raft_consensus.cc:740] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.109063 23786 consensus_queue.cc:260] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.110098 23786 ts_tablet_manager.cc:1434] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:32.110110 23787 ts_tablet_manager.cc:1434] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:32.110404 23787 tablet_bootstrap.cc:492] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.110438 23786 tablet_bootstrap.cc:492] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.112150 23786 tablet_bootstrap.cc:654] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.112568 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:32.112861 23787 tablet_bootstrap.cc:654] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.113312 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:32.115636 23217 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:32.090906 23790 tablet_bootstrap.cc:492] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.117295 23790 ts_tablet_manager.cc:1403] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.000s
W20251023 20:15:32.118289 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 84445.2us
W20251023 20:15:32.118564 23333 consensus_peers.cc:597] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbacaf2fea24405caaf84d59b625fca3. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:32.118726 23790 raft_consensus.cc:359] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.118912 23790 raft_consensus.cc:385] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.119009 23790 raft_consensus.cc:740] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.119235 23790 consensus_queue.cc:260] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.119401 25068 ts_tablet_manager.cc:1929] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.119659 25068 log.cc:1199] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/58ae6a6e58274b8baf952eaaefde7127
I20251023 20:15:32.119786 23790 ts_tablet_manager.cc:1434] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251023 20:15:32.119942 23333 leader_election.cc:343] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.120100 23790 tablet_bootstrap.cc:492] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.120594 25068 ts_tablet_manager.cc:1950] T 58ae6a6e58274b8baf952eaaefde7127 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:32.122107 23790 tablet_bootstrap.cc:654] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.123045 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 354) took 1214 ms. Trace:
I20251023 20:15:32.120065 23333 leader_election.cc:304] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.123687 25068 rpcz_store.cc:276] 1023 20:15:30.908564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.908610 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:32.066127 (+1157517us) ts_tablet_manager.cc:1151] Deleting tablet 58ae6a6e58274b8baf952eaaefde7127
1023 20:15:32.066139 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.100255 (+ 34116us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.119151 (+ 18896us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.122835 (+ 3684us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1157338}
I20251023 20:15:32.124729 25061 raft_consensus.cc:493] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.124878 25061 raft_consensus.cc:515] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.125522 25061 leader_election.cc:290] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.125550 25095 raft_consensus.cc:2749] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:32.126150 23333 leader_election.cc:343] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.126494 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:32.127031 23605 leader_election.cc:343] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.127063 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 58ae6a6e58274b8baf952eaaefde7127 (table default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]) successfully deleted
I20251023 20:15:32.127038 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 355) took 1217 ms. Trace:
I20251023 20:15:32.127492 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.127941 23604 leader_election.cc:343] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.128036 23604 leader_election.cc:304] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.128410 25061 raft_consensus.cc:2749] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.127588 25068 rpcz_store.cc:276] 1023 20:15:30.909028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.909072 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:32.125535 (+1216463us) ts_tablet_manager.cc:1151] Deleting tablet db758a7a7a89410f8ac1da72048fbc56
1023 20:15:32.125547 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.126840 (+ 1293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1181922}
W20251023 20:15:32.129420 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
I20251023 20:15:32.129761 24780 ts_tablet_manager.cc:1929] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.135581 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 358) took 1191 ms. Trace:
I20251023 20:15:32.136108 25068 rpcz_store.cc:276] 1023 20:15:30.943894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.943942 (+ 48us) service_pool.cc:225] Handling call
1023 20:15:32.130147 (+1186205us) ts_tablet_manager.cc:1151] Deleting tablet 252b5480222f49439052c86a4115cc9b
1023 20:15:32.130158 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.135554 (+ 5396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1185849}
W20251023 20:15:32.137158 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 252b5480222f49439052c86a4115cc9b because the tablet was not found. No further retry: Not found: Tablet not found: 252b5480222f49439052c86a4115cc9b
I20251023 20:15:32.138278 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 359) took 1194 ms. Trace:
I20251023 20:15:32.138574 25068 rpcz_store.cc:276] 1023 20:15:30.944019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.944167 (+ 148us) service_pool.cc:225] Handling call
1023 20:15:32.137762 (+1193595us) ts_tablet_manager.cc:1151] Deleting tablet 5acdc66df9c748a1a6300962f62fcf36
1023 20:15:32.137774 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.138168 (+ 394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1193394}
W20251023 20:15:32.139564 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5acdc66df9c748a1a6300962f62fcf36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5acdc66df9c748a1a6300962f62fcf36 already in progress: creating tablet
I20251023 20:15:32.140300 25068 tablet_replica.cc:333] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:32.140947 25068 raft_consensus.cc:2243] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.141722 25068 raft_consensus.cc:2272] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.144407 25068 ts_tablet_manager.cc:1916] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.142346 25095 consensus_queue.cc:1048] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:32.143138 23333 leader_election.cc:343] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:32.147594 23333 leader_election.cc:343] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.147869 25142 raft_consensus.cc:493] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.148025 25142 raft_consensus.cc:515] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.148181 24780 log.cc:1199] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:32.148638 25142 leader_election.cc:290] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.148651 24780 ts_tablet_manager.cc:1950] T 66a2f3adb5d347a0adc198cd1ef93304 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.149003 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 144) took 3386 ms. Trace:
I20251023 20:15:32.149129 24780 rpcz_store.cc:276] 1023 20:15:28.762874 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.762992 (+ 118us) service_pool.cc:225] Handling call
1023 20:15:32.070171 (+3307179us) ts_tablet_manager.cc:1151] Deleting tablet 66a2f3adb5d347a0adc198cd1ef93304
1023 20:15:32.070180 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.103052 (+ 32872us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.129570 (+ 26518us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.148970 (+ 19400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3307007}
I20251023 20:15:32.149454 24780 tablet_replica.cc:333] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:32.149896 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a630b83f614447ebf2a08851d6b827d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.150092 23670 raft_consensus.cc:2468] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:32.150568 23470 leader_election.cc:304] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.150909 25142 raft_consensus.cc:2804] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.150979 25142 raft_consensus.cc:493] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.151034 25142 raft_consensus.cc:3060] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.153555 25142 raft_consensus.cc:515] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.154738 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a630b83f614447ebf2a08851d6b827d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.154950 23670 raft_consensus.cc:3060] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.157552 23670 raft_consensus.cc:2468] T 8a630b83f614447ebf2a08851d6b827d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.158110 23470 leader_election.cc:304] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.158464 25167 raft_consensus.cc:2804] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.158588 25167 raft_consensus.cc:697] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:32.158843 25167 consensus_queue.cc:237] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.159598 25061 raft_consensus.cc:493] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.159716 25061 raft_consensus.cc:515] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.160260 25061 leader_election.cc:290] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.160980 25142 leader_election.cc:290] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.161343 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.161863 23604 leader_election.cc:343] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.158072 24297 consensus_queue.cc:1048] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.159725 25095 consensus_queue.cc:1048] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.161144 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.166129 24780 raft_consensus.cc:2243] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.161362 23786 tablet_bootstrap.cc:492] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
W20251023 20:15:32.169317 23605 leader_election.cc:343] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.169461 23605 leader_election.cc:304] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.169917 25061 raft_consensus.cc:2749] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.172945 23787 tablet_bootstrap.cc:492] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.173168 23787 ts_tablet_manager.cc:1403] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.006s
I20251023 20:15:32.173913 23787 raft_consensus.cc:359] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.174154 23787 raft_consensus.cc:385] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.174237 23787 raft_consensus.cc:740] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.174499 23787 consensus_queue.cc:260] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.174957 23786 ts_tablet_manager.cc:1403] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.002s
I20251023 20:15:32.176309 24780 raft_consensus.cc:2272] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.179833 23787 ts_tablet_manager.cc:1434] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251023 20:15:32.180193 23787 tablet_bootstrap.cc:492] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.181768 23787 tablet_bootstrap.cc:654] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.186771 23787 tablet_bootstrap.cc:492] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.188027 23787 ts_tablet_manager.cc:1403] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251023 20:15:32.188978 23787 raft_consensus.cc:359] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.189216 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.189635 25142 raft_consensus.cc:493] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.189767 25142 raft_consensus.cc:515] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.190347 25142 leader_election.cc:290] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.191078 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.193109 23668 raft_consensus.cc:2468] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.193485 23787 raft_consensus.cc:385] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.173736 25068 ts_tablet_manager.cc:1929] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:32.193687 23787 raft_consensus.cc:740] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.193830 25068 log.cc:1199] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/25079532547d4175802b84877790662d
W20251023 20:15:32.194229 23470 leader_election.cc:343] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: bb475e303d07491bbc19dc3f628bceb7
I20251023 20:15:32.194222 23787 consensus_queue.cc:260] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.194447 25068 ts_tablet_manager.cc:1950] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:32.194666 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 7645c36c983f438f8b1223256fd3ad2c (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]) successfully deleted
I20251023 20:15:32.195016 23668 raft_consensus.cc:1240] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 2 FOLLOWER]: Rejecting Update request from peer d220aba7c9c0400398ffde917376ee48 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251023 20:15:32.195242 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 360) took 1251 ms. Trace:
I20251023 20:15:32.195379 25068 rpcz_store.cc:276] 1023 20:15:30.944105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.944379 (+ 274us) service_pool.cc:225] Handling call
1023 20:15:32.140152 (+1195773us) ts_tablet_manager.cc:1151] Deleting tablet 25079532547d4175802b84877790662d
1023 20:15:32.140164 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.161404 (+ 21240us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.173338 (+ 11934us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.195223 (+ 21885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195493}
I20251023 20:15:32.196020 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 25079532547d4175802b84877790662d (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]) successfully deleted
I20251023 20:15:32.196496 23787 ts_tablet_manager.cc:1434] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251023 20:15:32.196580 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 361) took 1252 ms. Trace:
I20251023 20:15:32.196719 25068 rpcz_store.cc:276] 1023 20:15:30.944292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.944352 (+ 60us) service_pool.cc:225] Handling call
1023 20:15:32.196365 (+1252013us) ts_tablet_manager.cc:1151] Deleting tablet b90810781f724c78935a33d868ffa27d
1023 20:15:32.196377 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.196559 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251754}
I20251023 20:15:32.197247 23787 tablet_bootstrap.cc:492] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:32.197343 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet b90810781f724c78935a33d868ffa27d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b90810781f724c78935a33d868ffa27d already in progress: creating tablet
I20251023 20:15:32.197829 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 362) took 1238 ms. Trace:
I20251023 20:15:32.197968 25068 rpcz_store.cc:276] 1023 20:15:30.959659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.959726 (+ 67us) service_pool.cc:225] Handling call
1023 20:15:32.197636 (+1237910us) ts_tablet_manager.cc:1151] Deleting tablet fbab01ecb27a426188689408079eaa01
1023 20:15:32.197672 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.197810 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195274}
W20251023 20:15:32.198549 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
I20251023 20:15:32.175702 23790 tablet_bootstrap.cc:492] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.199287 23790 ts_tablet_manager.cc:1403] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.001s
I20251023 20:15:32.199790 23787 tablet_bootstrap.cc:654] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.200320 23790 raft_consensus.cc:359] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.200564 23790 raft_consensus.cc:385] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.200690 23790 raft_consensus.cc:740] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.200991 23790 consensus_queue.cc:260] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.201582 23790 ts_tablet_manager.cc:1434] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:32.202407 23790 tablet_bootstrap.cc:492] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.201386 23786 raft_consensus.cc:359] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.202826 23786 raft_consensus.cc:385] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.203015 23786 raft_consensus.cc:740] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.203445 23786 consensus_queue.cc:260] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.204144 23790 tablet_bootstrap.cc:654] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.204388 23786 ts_tablet_manager.cc:1434] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.028s user 0.001s sys 0.002s
I20251023 20:15:32.204910 23786 tablet_bootstrap.cc:492] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.207705 25061 raft_consensus.cc:493] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.207863 25061 raft_consensus.cc:515] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.208482 25061 leader_election.cc:290] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.209192 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882f7bf08e2f4b5382702c3d6eaf201e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.209725 23604 leader_election.cc:343] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.202883 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 363) took 1200 ms. Trace:
I20251023 20:15:32.210438 25068 rpcz_store.cc:276] 1023 20:15:31.002255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.002390 (+ 135us) service_pool.cc:225] Handling call
1023 20:15:32.198871 (+1196481us) ts_tablet_manager.cc:1151] Deleting tablet ea0cbd6bec674cf5831486f500cc7b36
1023 20:15:32.200855 (+ 1984us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.202863 (+ 2008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189182}
W20251023 20:15:32.212154 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
I20251023 20:15:32.213406 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 356) took 1270 ms. Trace:
I20251023 20:15:32.206450 24780 ts_tablet_manager.cc:1916] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.209548 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882f7bf08e2f4b5382702c3d6eaf201e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:32.214778 23605 leader_election.cc:343] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.214915 23605 leader_election.cc:304] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.213984 25068 rpcz_store.cc:276] 1023 20:15:30.942892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.942960 (+ 68us) service_pool.cc:225] Handling call
1023 20:15:32.212873 (+1269913us) ts_tablet_manager.cc:1151] Deleting tablet a708cd011802458aa0e2991fd4b4a2d3
1023 20:15:32.212887 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.213384 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1194990}
I20251023 20:15:32.215309 25061 raft_consensus.cc:2749] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:32.215559 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a708cd011802458aa0e2991fd4b4a2d3 because the tablet was not found. No further retry: Not found: Tablet not found: a708cd011802458aa0e2991fd4b4a2d3
I20251023 20:15:32.216975 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 357) took 1273 ms. Trace:
I20251023 20:15:32.217727 25068 rpcz_store.cc:276] 1023 20:15:30.943789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.944626 (+ 837us) service_pool.cc:225] Handling call
1023 20:15:32.216197 (+1271571us) ts_tablet_manager.cc:1151] Deleting tablet 15ced74d0dd74e5195ed98358957d0dd
1023 20:15:32.216428 (+ 231us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.216953 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196126}
I20251023 20:15:32.218986 25061 raft_consensus.cc:493] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.219142 25061 raft_consensus.cc:515] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:32.219401 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 15ced74d0dd74e5195ed98358957d0dd because the tablet was not found. No further retry: Not found: Tablet not found: 15ced74d0dd74e5195ed98358957d0dd
I20251023 20:15:32.221555 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 365) took 1203 ms. Trace:
I20251023 20:15:32.221885 25061 leader_election.cc:290] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.222591 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5af107f9a144edb9080b7135ab8aeb9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.222720 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5af107f9a144edb9080b7135ab8aeb9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.222815 23396 raft_consensus.cc:2468] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
W20251023 20:15:32.223351 23604 leader_election.cc:343] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.223583 23605 leader_election.cc:304] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.224049 25061 raft_consensus.cc:2804] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.224119 25061 raft_consensus.cc:493] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.224174 25061 raft_consensus.cc:3060] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.225873 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:32.226794 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 66a2f3adb5d347a0adc198cd1ef93304 (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]) successfully deleted
I20251023 20:15:32.227001 25061 raft_consensus.cc:515] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.228145 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5af107f9a144edb9080b7135ab8aeb9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:32.228680 23604 leader_election.cc:343] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.229453 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5af107f9a144edb9080b7135ab8aeb9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.229694 23396 raft_consensus.cc:3060] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.230109 25061 leader_election.cc:290] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.230461 23446 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:32.232463 23396 raft_consensus.cc:2468] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
I20251023 20:15:32.233108 23605 leader_election.cc:304] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.234973 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 955ec4740c4b4d3fa19a20faeaebc09e (DEFAULT_TABLE table=default.Table_aeba123341a14c5aba5a1ccca37e5395 [id=5b8f061cbf014eed80a09b13ee85429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.235234 25061 raft_consensus.cc:2804] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.235389 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955ec4740c4b4d3fa19a20faeaebc09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.235491 25061 raft_consensus.cc:697] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:32.235787 25061 consensus_queue.cc:237] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.236042 24821 raft_consensus.cc:493] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.236245 24821 raft_consensus.cc:515] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.236824 24821 leader_election.cc:290] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.237782 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 955ec4740c4b4d3fa19a20faeaebc09e (DEFAULT_TABLE table=default.Table_aeba123341a14c5aba5a1ccca37e5395 [id=5b8f061cbf014eed80a09b13ee85429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.238170 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955ec4740c4b4d3fa19a20faeaebc09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.241566 23397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fb01b3f4ff7f41e98063ebf76abeace6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.241647 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a86a25659ec44d7c825783036342ed45" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.241813 23397 raft_consensus.cc:3060] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.242115 23395 raft_consensus.cc:3060] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.244524 23395 raft_consensus.cc:2468] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.246686 23397 raft_consensus.cc:2468] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.246824 23388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a630b83f614447ebf2a08851d6b827d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.246971 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.247018 23388 raft_consensus.cc:3060] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.247710 25167 raft_consensus.cc:493] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.247828 25167 raft_consensus.cc:515] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.248220 25140 raft_consensus.cc:493] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.248358 25167 leader_election.cc:290] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.248353 25140 raft_consensus.cc:515] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.248893 25140 leader_election.cc:290] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:32.249198 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.249248 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "165d62dd42da460faccec706f08272a1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.249256 23395 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e194d03a6aa946fbb2c79d91b196440b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.249361 23668 raft_consensus.cc:2468] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:32.249492 23388 raft_consensus.cc:2468] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.249985 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e194d03a6aa946fbb2c79d91b196440b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.250217 23394 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ed0aa7e36e64843ad6b2ad4b415832a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.250398 23394 raft_consensus.cc:3060] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:32.250475 23604 leader_election.cc:343] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:32.250964 23605 leader_election.cc:343] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.250939 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e4189623c7a4ca1bdf73f71663e2b77" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.251057 23605 leader_election.cc:304] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.251097 23392 raft_consensus.cc:3060] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.251469 25140 raft_consensus.cc:2749] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.251736 23470 leader_election.cc:304] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.251983 25167 raft_consensus.cc:2804] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.252058 25167 raft_consensus.cc:493] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.252118 25167 raft_consensus.cc:3060] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.253592 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a630b83f614447ebf2a08851d6b827d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.253772 23389 raft_consensus.cc:2376] T 8a630b83f614447ebf2a08851d6b827d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1. Re-sending same reply.
I20251023 20:15:32.254043 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb475e303d07491bbc19dc3f628bceb7" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.254207 23387 raft_consensus.cc:2468] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:32.254550 25167 raft_consensus.cc:515] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.250711 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e4189623c7a4ca1bdf73f71663e2b77" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.255112 25167 leader_election.cc:290] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.256006 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.256160 23668 raft_consensus.cc:3060] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.257699 23394 raft_consensus.cc:2468] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.258596 23668 raft_consensus.cc:2468] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.258807 23392 raft_consensus.cc:2468] T 6e4189623c7a4ca1bdf73f71663e2b77 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.259157 23470 leader_election.cc:304] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.259478 25167 raft_consensus.cc:2804] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.259625 25167 raft_consensus.cc:697] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:32.259868 25167 consensus_queue.cc:237] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.262924 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25079532547d4175802b84877790662d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.263128 23396 raft_consensus.cc:3060] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.265993 23396 raft_consensus.cc:2468] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.271222 24821 raft_consensus.cc:493] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.271476 24821 raft_consensus.cc:515] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.272405 24821 leader_election.cc:290] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.276748 23511 tablet_service.cc:1505] Processing CreateTablet for tablet 9ea980533a5241f5bde89c7cc4c72f8d (DEFAULT_TABLE table=default.Table_9eb13d4c90b44054a9f549bd289a9c6e [id=f97ed62dc77d4f8991056b5268dd88df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.277419 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 9ea980533a5241f5bde89c7cc4c72f8d (DEFAULT_TABLE table=default.Table_9eb13d4c90b44054a9f549bd289a9c6e [id=f97ed62dc77d4f8991056b5268dd88df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.277555 23516 tablet_service.cc:1505] Processing CreateTablet for tablet 9047d641796b4f20b0de82b277ce877c (DEFAULT_TABLE table=default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.277838 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea980533a5241f5bde89c7cc4c72f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.277952 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9047d641796b4f20b0de82b277ce877c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.281581 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea980533a5241f5bde89c7cc4c72f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.287045 24780 ts_tablet_manager.cc:1929] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:32.287256 24780 log.cc:1199] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/0858358aabff47a69dc77f4669b225c5
I20251023 20:15:32.222155 25068 rpcz_store.cc:276] 1023 20:15:31.017577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.017621 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:32.220593 (+1202972us) ts_tablet_manager.cc:1151] Deleting tablet f908f086946949829cac15418a584730
1023 20:15:32.220605 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.221262 (+ 657us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199321}
I20251023 20:15:32.287591 24780 ts_tablet_manager.cc:1950] T 0858358aabff47a69dc77f4669b225c5 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.288318 23786 tablet_bootstrap.cc:654] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.288878 23534 raft_consensus.cc:1240] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 LEADER]: Rejecting Update request from peer d220aba7c9c0400398ffde917376ee48 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20251023 20:15:32.293643 25068 tablet_replica.cc:333] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:32.293992 25068 raft_consensus.cc:2243] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:32.294435 25068 pending_rounds.cc:70] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Trying to abort 1 pending ops.
I20251023 20:15:32.294555 25068 pending_rounds.cc:77] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082174416621568 op_type: NO_OP noop_request { }
I20251023 20:15:32.294745 25068 raft_consensus.cc:2889] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:32.294909 25068 raft_consensus.cc:2272] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.295804 25068 ts_tablet_manager.cc:1916] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.297773 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a570f339890946b7ab20c15de68462d5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.298018 23670 raft_consensus.cc:2468] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.298940 24821 raft_consensus.cc:493] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.300590 23362 tablet_service.cc:1505] Processing CreateTablet for tablet 955ec4740c4b4d3fa19a20faeaebc09e (DEFAULT_TABLE table=default.Table_aeba123341a14c5aba5a1ccca37e5395 [id=5b8f061cbf014eed80a09b13ee85429f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.302518 24821 raft_consensus.cc:515] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.304488 25167 raft_consensus.cc:493] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.304634 25167 raft_consensus.cc:515] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.305277 25167 leader_election.cc:290] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.305471 23471 leader_election.cc:304] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:32.306994 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a570f339890946b7ab20c15de68462d5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.307174 23530 raft_consensus.cc:2468] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.301019 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 9ea980533a5241f5bde89c7cc4c72f8d (DEFAULT_TABLE table=default.Table_9eb13d4c90b44054a9f549bd289a9c6e [id=f97ed62dc77d4f8991056b5268dd88df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.301647 25095 raft_consensus.cc:493] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.302773 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955ec4740c4b4d3fa19a20faeaebc09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.309054 25095 raft_consensus.cc:515] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.309501 24821 leader_election.cc:290] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.309679 25095 leader_election.cc:290] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.309775 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.309974 23670 raft_consensus.cc:2393] T eb5a6d9a4509431f97b69a3b138b5829 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:32.310500 25167 raft_consensus.cc:2804] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.310714 25167 raft_consensus.cc:697] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:32.310990 25167 consensus_queue.cc:237] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.312847 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 145) took 3549 ms. Trace:
I20251023 20:15:32.300675 23333 leader_election.cc:304] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.315654 25095 raft_consensus.cc:2804] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.315742 25095 raft_consensus.cc:493] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.315819 25095 raft_consensus.cc:3060] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.318392 25095 raft_consensus.cc:515] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.318928 25095 leader_election.cc:290] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.315747 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea980533a5241f5bde89c7cc4c72f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.324378 25061 raft_consensus.cc:493] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.324512 25061 raft_consensus.cc:515] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.325156 25061 leader_election.cc:290] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.325556 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b593b020b34a969f77d2f68178e89d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.326195 23604 leader_election.cc:343] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.327529 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b593b020b34a969f77d2f68178e89d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:32.328085 23605 leader_election.cc:343] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.328223 23605 leader_election.cc:304] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.328715 25061 raft_consensus.cc:2749] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.315826 24821 consensus_queue.cc:1059] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20251023 20:15:32.329890 24821 raft_consensus.cc:3055] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Stepping down as leader of term 1
I20251023 20:15:32.330005 24821 raft_consensus.cc:740] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.330309 24821 consensus_queue.cc:260] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.330677 24821 raft_consensus.cc:3060] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:32.332202 23787 tablet_bootstrap.cc:492] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.332343 23787 ts_tablet_manager.cc:1403] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.135s user 0.030s sys 0.054s
I20251023 20:15:32.333673 23787 raft_consensus.cc:359] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.333845 23787 raft_consensus.cc:385] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.333913 23787 raft_consensus.cc:740] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.334101 23787 consensus_queue.cc:260] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.334568 23787 ts_tablet_manager.cc:1434] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:32.334959 23787 tablet_bootstrap.cc:492] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.336599 23787 tablet_bootstrap.cc:654] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.339385 23790 tablet_bootstrap.cc:492] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:32.340229 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet f908f086946949829cac15418a584730 because the tablet was not found. No further retry: Not found: Tablet not found: f908f086946949829cac15418a584730
I20251023 20:15:32.341430 23786 tablet_bootstrap.cc:492] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.342027 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:32.342478 23786 ts_tablet_manager.cc:1403] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.001s
I20251023 20:15:32.350376 23790 ts_tablet_manager.cc:1403] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.148s user 0.004s sys 0.002s
I20251023 20:15:32.350960 23787 tablet_bootstrap.cc:492] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.351104 23787 ts_tablet_manager.cc:1403] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20251023 20:15:32.351109 23786 raft_consensus.cc:359] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.351243 23790 raft_consensus.cc:359] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.351418 23790 raft_consensus.cc:385] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.351475 23790 raft_consensus.cc:740] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.351456 23786 raft_consensus.cc:385] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.351614 23786 raft_consensus.cc:740] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.351671 23790 consensus_queue.cc:260] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.351926 23786 consensus_queue.cc:260] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.352062 23787 raft_consensus.cc:359] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.352227 23787 raft_consensus.cc:385] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.352250 23790 ts_tablet_manager.cc:1434] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:32.352285 23787 raft_consensus.cc:740] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.352509 23790 tablet_bootstrap.cc:492] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.352564 23787 consensus_queue.cc:260] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.353199 23787 ts_tablet_manager.cc:1434] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:32.353472 23787 tablet_bootstrap.cc:492] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.354362 23790 tablet_bootstrap.cc:654] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.355687 25172 consensus_queue.cc:1048] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.356629 23333 leader_election.cc:304] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.356904 25172 raft_consensus.cc:2804] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.356976 25172 raft_consensus.cc:493] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.357033 25172 raft_consensus.cc:3060] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.357039 23650 tablet_service.cc:1505] Processing CreateTablet for tablet 9047d641796b4f20b0de82b277ce877c (DEFAULT_TABLE table=default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.359608 23650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9047d641796b4f20b0de82b277ce877c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.359969 25172 raft_consensus.cc:515] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.360671 25172 leader_election.cc:290] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.364398 23786 ts_tablet_manager.cc:1434] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20251023 20:15:32.368342 23787 tablet_bootstrap.cc:654] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.369167 23786 tablet_bootstrap.cc:492] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.370587 23786 tablet_bootstrap.cc:654] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.371456 23633 tablet_service.cc:1505] Processing CreateTablet for tablet ee6464931f6f470ea73c8f2cdd3ed9da (DEFAULT_TABLE table=default.Table_4ad8e8f4f67e4639bcab0f4fd2c7b099 [id=94fc73e172904419b94d749aa2ef8dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.371882 23633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6464931f6f470ea73c8f2cdd3ed9da. 1 dirs total, 0 dirs full, 0 dirs failed
W20251023 20:15:32.374171 23333 leader_election.cc:343] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.377741 23631 tablet_service.cc:1505] Processing CreateTablet for tablet 29f5849dd3a949a98a415ef9e8076358 (DEFAULT_TABLE table=default.Table_7417d01d91d44665a6f021a647db582c [id=580458858c744a5f98da3fa134aea223]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.378177 23631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f5849dd3a949a98a415ef9e8076358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.378415 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 18939c8f489a4cca833ee3e15b07d71b (DEFAULT_TABLE table=default.Table_fbff76a175884e39a9efd67edd4db37e [id=9ec5b95ed1e340a784ea0a4cbf6e2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.378882 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 29f5849dd3a949a98a415ef9e8076358 (DEFAULT_TABLE table=default.Table_7417d01d91d44665a6f021a647db582c [id=580458858c744a5f98da3fa134aea223]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.379031 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18939c8f489a4cca833ee3e15b07d71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.379186 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f5849dd3a949a98a415ef9e8076358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.377741 23632 tablet_service.cc:1505] Processing CreateTablet for tablet 18939c8f489a4cca833ee3e15b07d71b (DEFAULT_TABLE table=default.Table_fbff76a175884e39a9efd67edd4db37e [id=9ec5b95ed1e340a784ea0a4cbf6e2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.381273 25172 raft_consensus.cc:493] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.370744 23516 tablet_service.cc:1505] Processing CreateTablet for tablet ee6464931f6f470ea73c8f2cdd3ed9da (DEFAULT_TABLE table=default.Table_4ad8e8f4f67e4639bcab0f4fd2c7b099 [id=94fc73e172904419b94d749aa2ef8dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.376873 23471 leader_election.cc:304] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:32.381428 25172 raft_consensus.cc:515] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.381634 23516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6464931f6f470ea73c8f2cdd3ed9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.382045 25172 leader_election.cc:290] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.382279 23471 leader_election.cc:304] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:32.383128 25142 raft_consensus.cc:2804] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.383244 25142 raft_consensus.cc:697] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:32.383471 25142 consensus_queue.cc:237] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.384583 24830 raft_consensus.cc:2804] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.384651 24830 raft_consensus.cc:493] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.384702 24830 raft_consensus.cc:3060] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.385607 23632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18939c8f489a4cca833ee3e15b07d71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.387045 24830 raft_consensus.cc:515] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.388818 24830 leader_election.cc:290] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.391119 24830 raft_consensus.cc:493] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.391314 24830 raft_consensus.cc:515] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.392395 24830 leader_election.cc:290] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.392489 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:32.396991 23470 leader_election.cc:343] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: bb475e303d07491bbc19dc3f628bceb7
I20251023 20:15:32.399788 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:32.400385 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:32.400767 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:32.405370 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:32.405956 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:32.406296 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:32.406735 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:32.409523 23471 leader_election.cc:304] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:32.410601 25167 raft_consensus.cc:2804] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.410781 25167 raft_consensus.cc:697] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:32.411072 25167 consensus_queue.cc:237] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.414032 23471 leader_election.cc:304] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:32.414222 23471 raft_consensus.cc:2686] T 25079532547d4175802b84877790662d P ea6853efbbcc484eb7529c7e9b37a47a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251023 20:15:32.418483 23530 raft_consensus.cc:3060] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.418738 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "514cb3b1ed024041ac160a7d9acff618" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.418926 23528 raft_consensus.cc:2468] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.420425 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.423676 23532 raft_consensus.cc:3060] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.425108 23786 tablet_bootstrap.cc:492] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.425762 23786 ts_tablet_manager.cc:1403] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.000s
I20251023 20:15:32.427232 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.427460 23670 raft_consensus.cc:2393] T 2a1bf4f01fbc4645bcf8392f56f853b2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:32.428755 23786 raft_consensus.cc:359] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.428815 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.429181 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.429347 23670 raft_consensus.cc:3060] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.429616 23786 raft_consensus.cc:385] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.429831 23786 raft_consensus.cc:740] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.430243 23786 consensus_queue.cc:260] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.431183 23786 ts_tablet_manager.cc:1434] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:32.431875 23786 tablet_bootstrap.cc:492] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.431973 23670 raft_consensus.cc:2468] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.432410 25068 ts_tablet_manager.cc:1929] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.432663 25068 log.cc:1199] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/ac55e14745534a02a2f6a422cab6b149
I20251023 20:15:32.433498 23790 tablet_bootstrap.cc:492] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.433647 23790 ts_tablet_manager.cc:1403] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.000s
I20251023 20:15:32.433660 23786 tablet_bootstrap.cc:654] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.437280 25068 ts_tablet_manager.cc:1950] T ac55e14745534a02a2f6a422cab6b149 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:32.439807 23787 tablet_bootstrap.cc:492] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.440006 23787 ts_tablet_manager.cc:1403] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.000s
I20251023 20:15:32.440855 23787 raft_consensus.cc:359] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.441116 23787 raft_consensus.cc:385] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.441207 23787 raft_consensus.cc:740] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.441461 23787 consensus_queue.cc:260] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.442216 25061 raft_consensus.cc:493] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.442327 25061 raft_consensus.cc:515] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.442919 25061 leader_election.cc:290] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.444283 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.444502 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:32.444841 23604 leader_election.cc:343] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.444885 23790 raft_consensus.cc:359] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.445139 23790 raft_consensus.cc:385] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:32.445220 23605 leader_election.cc:343] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.445284 23790 raft_consensus.cc:740] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.445312 23605 leader_election.cc:304] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.445487 23790 consensus_queue.cc:260] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.445749 25061 raft_consensus.cc:2749] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.448434 23787 ts_tablet_manager.cc:1434] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251023 20:15:32.448861 23787 tablet_bootstrap.cc:492] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.451030 23787 tablet_bootstrap.cc:654] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.456815 23786 tablet_bootstrap.cc:492] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.457056 23786 ts_tablet_manager.cc:1403] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251023 20:15:32.457871 23786 raft_consensus.cc:359] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.458190 23786 raft_consensus.cc:385] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.458284 23786 raft_consensus.cc:740] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.458544 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a570f339890946b7ab20c15de68462d5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.458572 23786 consensus_queue.cc:260] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.458724 23530 raft_consensus.cc:3060] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.460280 23790 ts_tablet_manager.cc:1434] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251023 20:15:32.460911 23786 ts_tablet_manager.cc:1434] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:32.461031 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 364) took 1448 ms. Trace:
I20251023 20:15:32.461639 23786 tablet_bootstrap.cc:492] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.461575 25068 rpcz_store.cc:276] 1023 20:15:31.012700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.012819 (+ 119us) service_pool.cc:225] Handling call
1023 20:15:32.293587 (+1280768us) ts_tablet_manager.cc:1151] Deleting tablet ac55e14745534a02a2f6a422cab6b149
1023 20:15:32.293597 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.364105 (+ 70508us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.432217 (+ 68112us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.461014 (+ 28797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1271914}
I20251023 20:15:32.462049 23530 raft_consensus.cc:2468] T a570f339890946b7ab20c15de68462d5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.462278 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet ac55e14745534a02a2f6a422cab6b149 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]) successfully deleted
I20251023 20:15:32.462822 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 371) took 1440 ms. Trace:
I20251023 20:15:32.463001 25068 rpcz_store.cc:276] 1023 20:15:31.022334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.022563 (+ 229us) service_pool.cc:225] Handling call
1023 20:15:32.462669 (+1440106us) ts_tablet_manager.cc:1151] Deleting tablet e418a62d00824cff89b37a7bace5c469
1023 20:15:32.462679 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.462804 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438909}
I20251023 20:15:32.463378 23786 tablet_bootstrap.cc:654] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:32.463661 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e418a62d00824cff89b37a7bace5c469 because the tablet was not found. No further retry: Not found: Tablet not found: e418a62d00824cff89b37a7bace5c469
I20251023 20:15:32.464185 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 367) took 1442 ms. Trace:
I20251023 20:15:32.464327 25068 rpcz_store.cc:276] 1023 20:15:31.021915 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.022627 (+ 712us) service_pool.cc:225] Handling call
1023 20:15:32.464001 (+1441374us) ts_tablet_manager.cc:1151] Deleting tablet 4b6c3e7169154e3497dd3570dca0d745
1023 20:15:32.464014 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.464143 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1440018}
W20251023 20:15:32.464831 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 4b6c3e7169154e3497dd3570dca0d745 because the tablet was not found. No further retry: Not found: Tablet not found: 4b6c3e7169154e3497dd3570dca0d745
I20251023 20:15:32.465241 25068 tablet_replica.cc:333] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:32.465937 25068 raft_consensus.cc:2243] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.467530 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a570f339890946b7ab20c15de68462d5" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.467733 23665 raft_consensus.cc:3060] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.467013 25068 raft_consensus.cc:2272] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.470387 23665 raft_consensus.cc:2468] T a570f339890946b7ab20c15de68462d5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.471405 24821 raft_consensus.cc:493] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.471546 24821 raft_consensus.cc:515] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.471751 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.471921 23396 raft_consensus.cc:2468] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:32.472095 24821 leader_election.cc:290] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.472170 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a1bf4f01fbc4645bcf8392f56f853b2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.472302 23399 raft_consensus.cc:3060] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.457595 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:32.472859 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:32.473023 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:32.473048 23368 tablet_service.cc:1505] Processing CreateTablet for tablet 9047d641796b4f20b0de82b277ce877c (DEFAULT_TABLE table=default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.473212 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:32.473464 23368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9047d641796b4f20b0de82b277ce877c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.473449 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:32.473744 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:32.473881 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:32.474133 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:32.474423 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:32.474591 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:32.474902 23399 raft_consensus.cc:2468] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.475025 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:32.475199 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb5a6d9a4509431f97b69a3b138b5829" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.475453 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:32.475538 23394 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb475e303d07491bbc19dc3f628bceb7" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:32.475672 23394 raft_consensus.cc:3060] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.475868 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:32.476017 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:32.476372 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:32.476507 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:32.476774 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:32.478119 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c05b9bcea6ff49188758271401270227" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.478649 23362 tablet_service.cc:1505] Processing CreateTablet for tablet ee6464931f6f470ea73c8f2cdd3ed9da (DEFAULT_TABLE table=default.Table_4ad8e8f4f67e4639bcab0f4fd2c7b099 [id=94fc73e172904419b94d749aa2ef8dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.478999 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6464931f6f470ea73c8f2cdd3ed9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.482152 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 18939c8f489a4cca833ee3e15b07d71b (DEFAULT_TABLE table=default.Table_fbff76a175884e39a9efd67edd4db37e [id=9ec5b95ed1e340a784ea0a4cbf6e2e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.482573 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18939c8f489a4cca833ee3e15b07d71b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.485440 23394 raft_consensus.cc:2468] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:32.488770 25068 ts_tablet_manager.cc:1916] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.489480 23367 tablet_service.cc:1505] Processing CreateTablet for tablet 29f5849dd3a949a98a415ef9e8076358 (DEFAULT_TABLE table=default.Table_7417d01d91d44665a6f021a647db582c [id=580458858c744a5f98da3fa134aea223]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:32.489935 23367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f5849dd3a949a98a415ef9e8076358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:32.490455 23790 tablet_bootstrap.cc:492] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.494480 23790 tablet_bootstrap.cc:654] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.498273 23786 tablet_bootstrap.cc:492] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.498514 23786 ts_tablet_manager.cc:1403] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.004s
I20251023 20:15:32.499362 23786 raft_consensus.cc:359] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.499619 23786 raft_consensus.cc:385] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.499720 23786 raft_consensus.cc:740] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.500029 23786 consensus_queue.cc:260] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.500701 23786 ts_tablet_manager.cc:1434] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:32.501593 23790 tablet_bootstrap.cc:492] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.501784 23790 ts_tablet_manager.cc:1403] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251023 20:15:32.502698 23790 raft_consensus.cc:359] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.502951 23790 raft_consensus.cc:385] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.503055 23790 raft_consensus.cc:740] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.503353 23790 consensus_queue.cc:260] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.504052 23790 ts_tablet_manager.cc:1434] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:32.504395 23786 tablet_bootstrap.cc:492] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.504705 23790 tablet_bootstrap.cc:492] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.506072 23790 tablet_bootstrap.cc:654] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.507723 23786 tablet_bootstrap.cc:654] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.510074 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edaed8c82da4e36a312477b15017460" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.510334 23530 raft_consensus.cc:2468] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.512463 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edaed8c82da4e36a312477b15017460" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.512542 25173 raft_consensus.cc:493] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.512655 23665 raft_consensus.cc:2468] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.512681 25173 raft_consensus.cc:515] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.513248 25173 leader_election.cc:290] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.520087 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "975fc4848c8f4db4bfd2dad9ac620d83" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.520274 23530 raft_consensus.cc:2468] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.521271 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "975fc4848c8f4db4bfd2dad9ac620d83" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.521768 23668 raft_consensus.cc:2468] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.524276 23786 tablet_bootstrap.cc:492] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.524405 23786 ts_tablet_manager.cc:1403] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20251023 20:15:32.525872 23790 tablet_bootstrap.cc:492] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.526083 23790 ts_tablet_manager.cc:1403] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20251023 20:15:32.534802 23787 tablet_bootstrap.cc:492] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.534945 23471 leader_election.cc:304] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:32.535076 23787 ts_tablet_manager.cc:1403] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.086s user 0.008s sys 0.001s
I20251023 20:15:32.535319 25167 raft_consensus.cc:3060] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.535866 23787 raft_consensus.cc:359] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.536015 23787 raft_consensus.cc:385] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.536056 23787 raft_consensus.cc:740] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.536324 23787 consensus_queue.cc:260] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.536950 23787 ts_tablet_manager.cc:1434] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:32.526860 23790 raft_consensus.cc:359] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.537102 23790 raft_consensus.cc:385] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.537184 23790 raft_consensus.cc:740] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.537415 23790 consensus_queue.cc:260] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.537933 25167 raft_consensus.cc:2749] T eb5a6d9a4509431f97b69a3b138b5829 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.538755 23471 leader_election.cc:304] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:32.539228 25167 raft_consensus.cc:2749] T c05b9bcea6ff49188758271401270227 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.540778 23790 ts_tablet_manager.cc:1434] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251023 20:15:32.540910 23471 leader_election.cc:304] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:32.541066 23471 raft_consensus.cc:2686] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251023 20:15:32.541179 23790 tablet_bootstrap.cc:492] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.541360 23787 tablet_bootstrap.cc:492] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.543037 23787 tablet_bootstrap.cc:654] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:32.543275 23333 leader_election.cc:343] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d9f73477a06542c0b215ae86716cdc4e
W20251023 20:15:32.545004 23333 leader_election.cc:343] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 514cb3b1ed024041ac160a7d9acff618
I20251023 20:15:32.547578 25068 ts_tablet_manager.cc:1929] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:32.547690 25068 log.cc:1199] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/bb475e303d07491bbc19dc3f628bceb7
I20251023 20:15:32.547979 25068 ts_tablet_manager.cc:1950] T bb475e303d07491bbc19dc3f628bceb7 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:32.548749 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 369) took 1526 ms. Trace:
I20251023 20:15:32.548794 23333 leader_election.cc:304] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.549435 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:32.549736 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:32.550060 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:32.550207 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:32.550411 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:32.550602 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:32.550760 23790 tablet_bootstrap.cc:654] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.553049 25040 raft_consensus.cc:2804] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.558531 25040 raft_consensus.cc:697] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.558833 25040 consensus_queue.cc:237] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.559244 25068 rpcz_store.cc:276] 1023 20:15:31.022107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.022346 (+ 239us) service_pool.cc:225] Handling call
1023 20:15:32.465129 (+1442783us) ts_tablet_manager.cc:1151] Deleting tablet bb475e303d07491bbc19dc3f628bceb7
1023 20:15:32.465175 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.515380 (+ 50205us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.547456 (+ 32076us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.548729 (+ 1273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441038}
I20251023 20:15:32.552878 25167 raft_consensus.cc:493] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.561316 25167 raft_consensus.cc:515] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.561949 25167 leader_election.cc:290] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.560801 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
W20251023 20:15:32.562839 23470 leader_election.cc:343] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 72408899f710470baaf6fdb268cb04c8
I20251023 20:15:32.563701 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 366) took 1541 ms. Trace:
I20251023 20:15:32.550997 23786 raft_consensus.cc:359] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.564690 23786 raft_consensus.cc:385] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.564916 23786 raft_consensus.cc:740] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.564347 25068 rpcz_store.cc:276] 1023 20:15:31.021777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.022708 (+ 931us) service_pool.cc:225] Handling call
1023 20:15:32.563295 (+1540587us) ts_tablet_manager.cc:1151] Deleting tablet 5e73ae95c73945f081c298f49d47c904
1023 20:15:32.563304 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.563683 (+ 379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533515}
I20251023 20:15:32.553483 23333 leader_election.cc:304] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.565538 25173 raft_consensus.cc:2804] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.565604 23333 leader_election.cc:304] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.565670 25173 raft_consensus.cc:697] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.565881 23786 consensus_queue.cc:260] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.565891 25173 consensus_queue.cc:237] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.566521 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 372) took 1543 ms. Trace:
I20251023 20:15:32.567251 25068 rpcz_store.cc:276] 1023 20:15:31.022459 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.022584 (+ 125us) service_pool.cc:225] Handling call
1023 20:15:32.565756 (+1543172us) ts_tablet_manager.cc:1151] Deleting tablet dafb9db710f94c328186e3179bf0b319
1023 20:15:32.565989 (+ 233us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.566281 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535884}
I20251023 20:15:32.569205 23786 ts_tablet_manager.cc:1434] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20251023 20:15:32.569200 24780 rpcz_store.cc:276] 1023 20:15:28.763336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.763544 (+ 208us) service_pool.cc:225] Handling call
1023 20:15:32.149392 (+3385848us) ts_tablet_manager.cc:1151] Deleting tablet 0858358aabff47a69dc77f4669b225c5
1023 20:15:32.149401 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.223092 (+ 73691us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.286916 (+ 63824us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.312777 (+ 25861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3385659}
I20251023 20:15:32.569530 23786 tablet_bootstrap.cc:492] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.569532 24780 tablet_replica.cc:333] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:32.569720 24780 raft_consensus.cc:2243] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.569832 25172 raft_consensus.cc:2804] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.569917 25172 raft_consensus.cc:493] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.569990 25172 raft_consensus.cc:3060] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.570396 23333 leader_election.cc:304] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.571689 24821 raft_consensus.cc:2804] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.571764 24821 raft_consensus.cc:493] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.571821 24821 raft_consensus.cc:3060] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.572650 25172 raft_consensus.cc:515] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.570250 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 370) took 1547 ms. Trace:
W20251023 20:15:32.572782 23333 leader_election.cc:343] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: d9f73477a06542c0b215ae86716cdc4e
I20251023 20:15:32.572890 23333 leader_election.cc:304] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.572858 25068 rpcz_store.cc:276] 1023 20:15:31.022239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.022539 (+ 300us) service_pool.cc:225] Handling call
1023 20:15:32.569074 (+1546535us) ts_tablet_manager.cc:1151] Deleting tablet 5964da70989b4e6cb77fe5d0a52ce9cf
1023 20:15:32.569370 (+ 296us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.569889 (+ 519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538975}
I20251023 20:15:32.573345 24297 raft_consensus.cc:2749] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.573640 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 373) took 1550 ms. Trace:
I20251023 20:15:32.573773 25068 rpcz_store.cc:276] 1023 20:15:31.023360 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.023415 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:32.573441 (+1550026us) ts_tablet_manager.cc:1151] Deleting tablet 3894dd4feeea4c16861a6c9520059474
1023 20:15:32.573475 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.573620 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542978}
I20251023 20:15:32.574304 25172 leader_election.cc:290] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.574260 24821 raft_consensus.cc:515] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.562721 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet bb475e303d07491bbc19dc3f628bceb7 (table default.Table_0d31043ac68d4a0394d7ba5b60d539c8 [id=0a9d3a3303d945f199d265637a859772]) successfully deleted
I20251023 20:15:32.574775 24821 leader_election.cc:290] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:32.574908 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5e73ae95c73945f081c298f49d47c904 because the tablet was not found. No further retry: Not found: Tablet not found: 5e73ae95c73945f081c298f49d47c904
I20251023 20:15:32.575006 23786 tablet_bootstrap.cc:654] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.574998 25095 raft_consensus.cc:493] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.575094 25095 raft_consensus.cc:515] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.550971 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:32.569984 24780 raft_consensus.cc:2272] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.575575 25095 leader_election.cc:290] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.573326 23333 leader_election.cc:304] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
W20251023 20:15:32.575937 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet dafb9db710f94c328186e3179bf0b319 because the tablet was not found. No further retry: Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319
I20251023 20:15:32.575976 24780 ts_tablet_manager.cc:1916] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:32.576167 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 5964da70989b4e6cb77fe5d0a52ce9cf because the tablet was not found. No further retry: Not found: Tablet not found: 5964da70989b4e6cb77fe5d0a52ce9cf
I20251023 20:15:32.576292 25095 raft_consensus.cc:2804] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.576354 25095 raft_consensus.cc:493] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.576345 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:32.576406 25095 raft_consensus.cc:3060] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:32.576406 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 3894dd4feeea4c16861a6c9520059474 because the tablet was not found. No further retry: Not found: Tablet not found: 3894dd4feeea4c16861a6c9520059474
I20251023 20:15:32.576835 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:32.577065 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:32.577734 23790 tablet_bootstrap.cc:492] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.577874 23790 ts_tablet_manager.cc:1403] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.002s
I20251023 20:15:32.578564 23790 raft_consensus.cc:359] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.578769 23790 raft_consensus.cc:385] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.577785 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:32.578913 23790 raft_consensus.cc:740] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.578773 25095 raft_consensus.cc:515] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.579195 23790 consensus_queue.cc:260] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.579502 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:32.579557 25095 leader_election.cc:290] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.579980 23333 leader_election.cc:304] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.580691 25095 raft_consensus.cc:2749] T 2a1bf4f01fbc4645bcf8392f56f853b2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.581418 25095 consensus_queue.cc:1048] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.582273 25095 consensus_queue.cc:1048] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.582021 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 374) took 1551 ms. Trace:
I20251023 20:15:32.582592 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
W20251023 20:15:32.582883 23333 leader_election.cc:343] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.583112 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:32.582684 23790 ts_tablet_manager.cc:1434] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:32.583437 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:32.583739 23790 tablet_bootstrap.cc:492] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.583770 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:32.584125 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:32.584069 23787 tablet_bootstrap.cc:492] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.585184 23787 ts_tablet_manager.cc:1403] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20251023 20:15:32.585223 23790 tablet_bootstrap.cc:654] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.585602 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:32.585944 23787 raft_consensus.cc:359] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.586118 23787 raft_consensus.cc:385] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.586179 23787 raft_consensus.cc:740] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.586289 23214 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:32.586424 23787 consensus_queue.cc:260] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.587066 23787 ts_tablet_manager.cc:1434] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:32.587446 23787 tablet_bootstrap.cc:492] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.582825 25068 rpcz_store.cc:276] 1023 20:15:31.030246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.030493 (+ 247us) service_pool.cc:225] Handling call
1023 20:15:32.580404 (+1549911us) ts_tablet_manager.cc:1151] Deleting tablet e0f8d8184a68453abd2792ccc7740aa1
1023 20:15:32.580914 (+ 510us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.581644 (+ 730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1549759}
W20251023 20:15:32.588976 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e0f8d8184a68453abd2792ccc7740aa1 because the tablet was not found. No further retry: Not found: Tablet not found: e0f8d8184a68453abd2792ccc7740aa1
W20251023 20:15:32.589995 23333 consensus_peers.cc:597] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 994a555b24e4432a84601c54d422aa5d. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:32.590690 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 383) took 1559 ms. Trace:
I20251023 20:15:32.591934 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385900c60d584c15b702f186ecfbc036" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.592125 23668 raft_consensus.cc:2468] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.592279 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "975fc4848c8f4db4bfd2dad9ac620d83" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.592435 23534 raft_consensus.cc:3060] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.591460 25068 rpcz_store.cc:276] 1023 20:15:31.031461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.031536 (+ 75us) service_pool.cc:225] Handling call
1023 20:15:32.589845 (+1558309us) ts_tablet_manager.cc:1151] Deleting tablet fe922c438a894a7aa04699b2d2897a28
1023 20:15:32.589857 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.590448 (+ 591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557927}
I20251023 20:15:32.593160 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 0858358aabff47a69dc77f4669b225c5 (table default.Table_c9ccf1142a9b449f91e99360376a7b22 [id=c36b140eefe1435d93b7ba48ca07e4a1]) successfully deleted
I20251023 20:15:32.594410 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 384) took 1562 ms. Trace:
I20251023 20:15:32.595093 23534 raft_consensus.cc:2468] T 975fc4848c8f4db4bfd2dad9ac620d83 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:32.593163 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet fe922c438a894a7aa04699b2d2897a28 because the tablet was not found. No further retry: Not found: Tablet not found: fe922c438a894a7aa04699b2d2897a28
I20251023 20:15:32.594461 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edaed8c82da4e36a312477b15017460" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.595338 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.595413 23528 raft_consensus.cc:3060] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.595649 23394 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72408899f710470baaf6fdb268cb04c8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.595844 23394 raft_consensus.cc:2393] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:32.595186 25068 rpcz_store.cc:276] 1023 20:15:31.031561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.031607 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:32.593627 (+1562020us) ts_tablet_manager.cc:1151] Deleting tablet 585a9af140d9479fb83d13e68a62300c
1023 20:15:32.593867 (+ 240us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.594389 (+ 522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561859}
W20251023 20:15:32.596446 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 585a9af140d9479fb83d13e68a62300c because the tablet was not found. No further retry: Not found: Tablet not found: 585a9af140d9479fb83d13e68a62300c
I20251023 20:15:32.596712 23787 tablet_bootstrap.cc:654] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.597728 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 388) took 1565 ms. Trace:
I20251023 20:15:32.599007 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385900c60d584c15b702f186ecfbc036" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.599174 23532 raft_consensus.cc:2468] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.598425 25068 rpcz_store.cc:276] 1023 20:15:31.031953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032014 (+ 61us) service_pool.cc:225] Handling call
1023 20:15:32.597203 (+1565189us) ts_tablet_manager.cc:1151] Deleting tablet 2fa6a838b1bd409ba2a4e2074576ee23
1023 20:15:32.597213 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.597710 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565032}
I20251023 20:15:32.599673 23333 leader_election.cc:304] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.600019 25095 raft_consensus.cc:2804] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251023 20:15:32.600035 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2fa6a838b1bd409ba2a4e2074576ee23 because the tablet was not found. No further retry: Not found: Tablet not found: 2fa6a838b1bd409ba2a4e2074576ee23
I20251023 20:15:32.600128 25095 raft_consensus.cc:493] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.600198 25095 raft_consensus.cc:3060] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.600457 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.600661 23662 raft_consensus.cc:2468] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.601063 23661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "975fc4848c8f4db4bfd2dad9ac620d83" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.601234 23661 raft_consensus.cc:3060] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.602173 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 377) took 1570 ms. Trace:
I20251023 20:15:32.602813 25095 raft_consensus.cc:515] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.603412 25095 leader_election.cc:290] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.603902 23661 raft_consensus.cc:2468] T 975fc4848c8f4db4bfd2dad9ac620d83 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.602941 25068 rpcz_store.cc:276] 1023 20:15:31.031009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032503 (+ 1494us) service_pool.cc:225] Handling call
1023 20:15:32.601368 (+1568865us) ts_tablet_manager.cc:1151] Deleting tablet 66a2f3adb5d347a0adc198cd1ef93304
1023 20:15:32.601697 (+ 329us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.601922 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1568447}
I20251023 20:15:32.604210 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1edaed8c82da4e36a312477b15017460" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.604351 23670 raft_consensus.cc:3060] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:32.604655 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 66a2f3adb5d347a0adc198cd1ef93304 because the tablet was not found. No further retry: Not found: Tablet not found: 66a2f3adb5d347a0adc198cd1ef93304
I20251023 20:15:32.605696 25095 consensus_queue.cc:1048] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.606702 23670 raft_consensus.cc:2468] T 1edaed8c82da4e36a312477b15017460 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.606762 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 368) took 1584 ms. Trace:
I20251023 20:15:32.607084 23333 leader_election.cc:304] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.607367 25095 raft_consensus.cc:2804] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.607458 25095 raft_consensus.cc:493] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.607546 25095 raft_consensus.cc:3060] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.607749 23471 leader_election.cc:304] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:32.607043 25068 rpcz_store.cc:276] 1023 20:15:31.022011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.022604 (+ 593us) service_pool.cc:225] Handling call
1023 20:15:32.605201 (+1582597us) ts_tablet_manager.cc:1151] Deleting tablet 925e53d9ea6f4c7baa38551eb917828d
1023 20:15:32.606602 (+ 1401us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.606701 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1572432}
I20251023 20:15:32.608659 25167 raft_consensus.cc:3060] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:32.608951 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 925e53d9ea6f4c7baa38551eb917828d because the tablet was not found. No further retry: Not found: Tablet not found: 925e53d9ea6f4c7baa38551eb917828d
I20251023 20:15:32.610555 25095 raft_consensus.cc:515] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.610227 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 376) took 1579 ms. Trace:
I20251023 20:15:32.611152 25095 leader_election.cc:290] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.611135 25167 raft_consensus.cc:2749] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.611286 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385900c60d584c15b702f186ecfbc036" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.611435 23530 raft_consensus.cc:3060] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.611755 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.611009 25068 rpcz_store.cc:276] 1023 20:15:31.030930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032789 (+ 1859us) service_pool.cc:225] Handling call
1023 20:15:32.609447 (+1576658us) ts_tablet_manager.cc:1151] Deleting tablet 7645c36c983f438f8b1223256fd3ad2c
1023 20:15:32.609677 (+ 230us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.609963 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576520}
I20251023 20:15:32.611933 23333 leader_election.cc:304] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:32.612838 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7645c36c983f438f8b1223256fd3ad2c because the tablet was not found. No further retry: Not found: Tablet not found: 7645c36c983f438f8b1223256fd3ad2c
W20251023 20:15:32.612927 23333 leader_election.cc:343] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:32.613379 23333 leader_election.cc:343] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:32.613523 23333 leader_election.cc:343] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 514cb3b1ed024041ac160a7d9acff618
I20251023 20:15:32.613770 23333 leader_election.cc:304] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.613919 23530 raft_consensus.cc:2468] T 385900c60d584c15b702f186ecfbc036 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.614557 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "514cb3b1ed024041ac160a7d9acff618" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.614209 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 375) took 1583 ms. Trace:
I20251023 20:15:32.614734 23532 raft_consensus.cc:3060] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.615064 25068 rpcz_store.cc:276] 1023 20:15:31.030795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032944 (+ 2149us) service_pool.cc:225] Handling call
1023 20:15:32.613455 (+1580511us) ts_tablet_manager.cc:1151] Deleting tablet e5368813164949ff80e27bf59dcc931a
1023 20:15:32.613633 (+ 178us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.614015 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1580374}
W20251023 20:15:32.616134 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet e5368813164949ff80e27bf59dcc931a because the tablet was not found. No further retry: Not found: Tablet not found: e5368813164949ff80e27bf59dcc931a
I20251023 20:15:32.616679 25040 raft_consensus.cc:2804] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.616798 25040 raft_consensus.cc:697] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.617051 25040 consensus_queue.cc:237] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.617554 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.617302 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 380) took 1585 ms. Trace:
I20251023 20:15:32.617733 23663 raft_consensus.cc:3060] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.618115 24821 raft_consensus.cc:2804] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.618221 24821 raft_consensus.cc:697] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.618433 24821 consensus_queue.cc:237] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.620302 23663 raft_consensus.cc:2468] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.597878 23528 raft_consensus.cc:2468] T 1edaed8c82da4e36a312477b15017460 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.621366 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "385900c60d584c15b702f186ecfbc036" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.621524 23670 raft_consensus.cc:3060] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.623894 23670 raft_consensus.cc:2468] T 385900c60d584c15b702f186ecfbc036 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.625156 25068 rpcz_store.cc:276] 1023 20:15:31.031247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.033111 (+ 1864us) service_pool.cc:225] Handling call
1023 20:15:32.616620 (+1583509us) ts_tablet_manager.cc:1151] Deleting tablet d0e61de7a93d450d9ed79eef82d7e5e3
1023 20:15:32.616783 (+ 163us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.617006 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583280}
I20251023 20:15:32.625689 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 379) took 1594 ms. Trace:
W20251023 20:15:32.626214 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet d0e61de7a93d450d9ed79eef82d7e5e3 because the tablet was not found. No further retry: Not found: Tablet not found: d0e61de7a93d450d9ed79eef82d7e5e3
I20251023 20:15:32.626758 25068 rpcz_store.cc:276] 1023 20:15:31.031165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.033366 (+ 2201us) service_pool.cc:225] Handling call
1023 20:15:32.625532 (+1592166us) ts_tablet_manager.cc:1151] Deleting tablet 23225d31c2e647398aa58494c627507e
1023 20:15:32.625545 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.625649 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592019}
I20251023 20:15:32.627120 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 378) took 1596 ms. Trace:
W20251023 20:15:32.627583 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 23225d31c2e647398aa58494c627507e because the tablet was not found. No further retry: Not found: Tablet not found: 23225d31c2e647398aa58494c627507e
I20251023 20:15:32.636555 25068 rpcz_store.cc:276] 1023 20:15:31.031083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.033534 (+ 2451us) service_pool.cc:225] Handling call
1023 20:15:32.627008 (+1593474us) ts_tablet_manager.cc:1151] Deleting tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf
1023 20:15:32.627016 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.627083 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593346}
I20251023 20:15:32.637187 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 387) took 1605 ms. Trace:
I20251023 20:15:32.637300 25068 rpcz_store.cc:276] 1023 20:15:31.031825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.031864 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:32.637050 (+1605186us) ts_tablet_manager.cc:1151] Deleting tablet 2532c8567db14901b383f42e5f04b4c2
1023 20:15:32.637060 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.637152 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602463}
I20251023 20:15:32.637586 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 386) took 1605 ms. Trace:
I20251023 20:15:32.637676 25068 rpcz_store.cc:276] 1023 20:15:31.031729 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.031784 (+ 55us) service_pool.cc:225] Handling call
1023 20:15:32.637496 (+1605712us) ts_tablet_manager.cc:1151] Deleting tablet 283166bbd8004ce3bca9f2ecbdbcbf48
1023 20:15:32.637503 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.637548 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602815}
I20251023 20:15:32.637921 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 385) took 1606 ms. Trace:
I20251023 20:15:32.638005 25068 rpcz_store.cc:276] 1023 20:15:31.031645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032292 (+ 647us) service_pool.cc:225] Handling call
1023 20:15:32.637844 (+1605552us) ts_tablet_manager.cc:1151] Deleting tablet a27f192792cf41288d69ca15ab538472
1023 20:15:32.637851 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.637894 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602770}
I20251023 20:15:32.638257 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 382) took 1606 ms. Trace:
I20251023 20:15:32.638351 25068 rpcz_store.cc:276] 1023 20:15:31.031395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032357 (+ 962us) service_pool.cc:225] Handling call
1023 20:15:32.638171 (+1605814us) ts_tablet_manager.cc:1151] Deleting tablet 40e0301fb5974ec5b6e8291230a736c2
1023 20:15:32.638178 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.638230 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602754}
I20251023 20:15:32.638615 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 389) took 1606 ms. Trace:
I20251023 20:15:32.638710 25068 rpcz_store.cc:276] 1023 20:15:31.032058 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032438 (+ 380us) service_pool.cc:225] Handling call
1023 20:15:32.638521 (+1606083us) ts_tablet_manager.cc:1151] Deleting tablet 86b9f4c5d9e0469c87071a33722ce3eb
1023 20:15:32.638529 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.638585 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602956}
I20251023 20:15:32.638731 23333 leader_election.cc:304] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:32.639055 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 381) took 1607 ms. Trace:
I20251023 20:15:32.639151 25068 rpcz_store.cc:276] 1023 20:15:31.031326 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.032469 (+ 1143us) service_pool.cc:225] Handling call
1023 20:15:32.638890 (+1606421us) ts_tablet_manager.cc:1151] Deleting tablet a21ec49194e24723b31fd9c86f1f85e4
1023 20:15:32.638898 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.638943 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1603199}
I20251023 20:15:32.639444 25068 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 390) took 1225 ms. Trace:
I20251023 20:15:32.639531 25068 rpcz_store.cc:276] 1023 20:15:31.413999 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.414057 (+ 58us) service_pool.cc:225] Handling call
1023 20:15:32.639335 (+1225278us) ts_tablet_manager.cc:1151] Deleting tablet 7f2a4f1d52984af58d6286671389b6a9
1023 20:15:32.639343 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.639418 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225053}
I20251023 20:15:32.639825 25095 raft_consensus.cc:2804] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.639950 25095 raft_consensus.cc:697] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.640183 25095 consensus_queue.cc:237] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:32.640336 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf because the tablet was not found. No further retry: Not found: Tablet not found: 1e8eb7603c3e46f1b15cdff7a7bbedcf
W20251023 20:15:32.640632 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 2532c8567db14901b383f42e5f04b4c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2532c8567db14901b383f42e5f04b4c2
W20251023 20:15:32.640827 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 because the tablet was not found. No further retry: Not found: Tablet not found: 283166bbd8004ce3bca9f2ecbdbcbf48
W20251023 20:15:32.641013 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a27f192792cf41288d69ca15ab538472 because the tablet was not found. No further retry: Not found: Tablet not found: a27f192792cf41288d69ca15ab538472
W20251023 20:15:32.641196 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 40e0301fb5974ec5b6e8291230a736c2 because the tablet was not found. No further retry: Not found: Tablet not found: 40e0301fb5974ec5b6e8291230a736c2
W20251023 20:15:32.641394 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 86b9f4c5d9e0469c87071a33722ce3eb because the tablet was not found. No further retry: Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb
W20251023 20:15:32.641559 23204 catalog_manager.cc:4952] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet a21ec49194e24723b31fd9c86f1f85e4 because the tablet was not found. No further retry: Not found: Tablet not found: a21ec49194e24723b31fd9c86f1f85e4
W20251023 20:15:32.641768 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 7f2a4f1d52984af58d6286671389b6a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f2a4f1d52984af58d6286671389b6a9 already in progress: creating tablet
I20251023 20:15:32.641855 23790 tablet_bootstrap.cc:492] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.641973 23790 ts_tablet_manager.cc:1403] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.006s
I20251023 20:15:32.642657 23790 raft_consensus.cc:359] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.642867 23790 raft_consensus.cc:385] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.642949 23790 raft_consensus.cc:740] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.643138 23790 consensus_queue.cc:260] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.643729 23790 ts_tablet_manager.cc:1434] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:32.644040 23790 tablet_bootstrap.cc:492] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.628001 23532 raft_consensus.cc:2468] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.628695 23786 tablet_bootstrap.cc:492] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.651701 23786 ts_tablet_manager.cc:1403] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20251023 20:15:32.654275 23333 leader_election.cc:304] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:32.654613 24822 raft_consensus.cc:2804] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.654737 24822 raft_consensus.cc:697] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.654714 23333 leader_election.cc:304] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.654990 24822 consensus_queue.cc:237] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.655920 24297 raft_consensus.cc:2804] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:32.656019 24297 raft_consensus.cc:697] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.656208 24297 consensus_queue.cc:237] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:32.656375 23203 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92c50c4845524163834861438291a3ec consensus state has no leader
I20251023 20:15:32.657586 23786 raft_consensus.cc:359] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.659185 23786 raft_consensus.cc:385] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.659794 23786 raft_consensus.cc:740] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.661129 23786 consensus_queue.cc:260] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.665766 23786 ts_tablet_manager.cc:1434] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20251023 20:15:32.668715 23786 tablet_bootstrap.cc:492] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.673527 24780 ts_tablet_manager.cc:1929] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.676365 24780 log.cc:1199] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e418a62d00824cff89b37a7bace5c469
I20251023 20:15:32.677175 24780 ts_tablet_manager.cc:1950] T e418a62d00824cff89b37a7bace5c469 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.677249 23786 tablet_bootstrap.cc:654] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.678473 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 146) took 3914 ms. Trace:
I20251023 20:15:32.678573 24780 rpcz_store.cc:276] 1023 20:15:28.763919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.764117 (+ 198us) service_pool.cc:225] Handling call
1023 20:15:32.569485 (+3805368us) ts_tablet_manager.cc:1151] Deleting tablet e418a62d00824cff89b37a7bace5c469
1023 20:15:32.569496 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.581706 (+ 12210us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.673390 (+ 91684us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.678453 (+ 5063us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3804629}
I20251023 20:15:32.679057 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 147) took 3914 ms. Trace:
I20251023 20:15:32.679137 24780 rpcz_store.cc:276] 1023 20:15:28.765017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.765179 (+ 162us) service_pool.cc:225] Handling call
1023 20:15:32.678924 (+3913745us) ts_tablet_manager.cc:1151] Deleting tablet db758a7a7a89410f8ac1da72048fbc56
1023 20:15:32.678934 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.679039 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3913558}
I20251023 20:15:32.679409 24780 tablet_replica.cc:333] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:32.681051 23787 tablet_bootstrap.cc:492] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.681213 23787 ts_tablet_manager.cc:1403] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.007s
I20251023 20:15:32.682564 24780 raft_consensus.cc:2243] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.682768 24780 raft_consensus.cc:2272] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.683323 24780 ts_tablet_manager.cc:1916] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.683619 23787 raft_consensus.cc:359] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.683812 23787 raft_consensus.cc:385] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.683859 23787 raft_consensus.cc:740] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.684049 23787 consensus_queue.cc:260] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.684510 23787 ts_tablet_manager.cc:1434] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:32.684813 23787 tablet_bootstrap.cc:492] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.686434 23787 tablet_bootstrap.cc:654] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.690562 25228 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 690ffde75cd745b6a87925967d54471a)
I20251023 20:15:32.691361 25228 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:32.707633 25228 leader_election.cc:290] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:32.710960 23787 tablet_bootstrap.cc:492] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.712522 23787 ts_tablet_manager.cc:1403] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.001s
I20251023 20:15:32.713670 23787 raft_consensus.cc:359] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.722080 23787 raft_consensus.cc:385] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.722244 23787 raft_consensus.cc:740] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.722564 23787 consensus_queue.cc:260] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.725293 23227 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 755 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a" is_pre_election: true
I20251023 20:15:32.726790 23787 ts_tablet_manager.cc:1434] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251023 20:15:32.727149 23787 tablet_bootstrap.cc:492] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.733539 23787 tablet_bootstrap.cc:654] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.736701 24780 ts_tablet_manager.cc:1929] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.736939 24780 log.cc:1199] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/1e8eb7603c3e46f1b15cdff7a7bbedcf
I20251023 20:15:32.742185 23790 tablet_bootstrap.cc:654] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.749729 25237 raft_consensus.cc:493] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.749979 25237 raft_consensus.cc:515] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.750716 25237 leader_election.cc:290] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.757331 23787 tablet_bootstrap.cc:492] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.757685 23787 ts_tablet_manager.cc:1403] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20251023 20:15:32.758598 23786 tablet_bootstrap.cc:492] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.758793 23786 ts_tablet_manager.cc:1403] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.001s
I20251023 20:15:32.759577 23786 raft_consensus.cc:359] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.759804 23786 raft_consensus.cc:385] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.759884 23786 raft_consensus.cc:740] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.759809 23787 raft_consensus.cc:359] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.760216 23787 raft_consensus.cc:385] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.760345 23787 raft_consensus.cc:740] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.760809 23787 consensus_queue.cc:260] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.764271 23786 consensus_queue.cc:260] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:32.770795 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 116088us
I20251023 20:15:32.777730 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d238e071cb4d49b173febd90bc909b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.777969 23662 raft_consensus.cc:2468] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.778985 24780 ts_tablet_manager.cc:1950] T 1e8eb7603c3e46f1b15cdff7a7bbedcf P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.780270 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 148) took 4014 ms. Trace:
I20251023 20:15:32.780278 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d238e071cb4d49b173febd90bc909b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.780555 23528 raft_consensus.cc:2468] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.783854 24780 rpcz_store.cc:276] 1023 20:15:28.765552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.765736 (+ 184us) service_pool.cc:225] Handling call
1023 20:15:32.679365 (+3913629us) ts_tablet_manager.cc:1151] Deleting tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf
1023 20:15:32.679375 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.733614 (+ 54239us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.736581 (+ 2967us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.780248 (+ 43667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3913481}
I20251023 20:15:32.784752 24780 tablet_replica.cc:333] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:32.784782 23787 ts_tablet_manager.cc:1434] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251023 20:15:32.785676 24780 raft_consensus.cc:2243] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:32.787132 24780 raft_consensus.cc:2272] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.787416 23787 tablet_bootstrap.cc:492] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.786660 23790 tablet_bootstrap.cc:492] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.792114 23790 ts_tablet_manager.cc:1403] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.148s user 0.000s sys 0.013s
I20251023 20:15:32.794037 23790 raft_consensus.cc:359] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.797394 23790 raft_consensus.cc:385] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.797658 23790 raft_consensus.cc:740] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.798066 23666 raft_consensus.cc:1275] T 165d62dd42da460faccec706f08272a1 P 9bc6522db11e4960a9aad84435371a32 [term 2 FOLLOWER]: Refusing update from remote peer ea6853efbbcc484eb7529c7e9b37a47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251023 20:15:32.794538 24780 ts_tablet_manager.cc:1916] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.798698 25167 consensus_queue.cc:1048] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.798008 23790 consensus_queue.cc:260] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.801625 23790 ts_tablet_manager.cc:1434] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:32.802379 23790 tablet_bootstrap.cc:492] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.804621 23790 tablet_bootstrap.cc:654] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.812933 25040 raft_consensus.cc:493] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.814668 25040 raft_consensus.cc:515] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.815332 25040 leader_election.cc:290] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.818477 23790 tablet_bootstrap.cc:492] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.819046 23790 ts_tablet_manager.cc:1403] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251023 20:15:32.820681 23790 raft_consensus.cc:359] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.824127 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet e418a62d00824cff89b37a7bace5c469 (table default.Table_3a893d487d5b46ca84e6d00ce6cf7137 [id=751ae1aba5564ffa8dd6274778d1b8c7]) successfully deleted
W20251023 20:15:32.824469 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet db758a7a7a89410f8ac1da72048fbc56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db758a7a7a89410f8ac1da72048fbc56 already in progress: creating tablet
I20251023 20:15:32.824484 23790 raft_consensus.cc:385] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.824769 23790 raft_consensus.cc:740] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.821561 24780 ts_tablet_manager.cc:1929] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.825199 24780 log.cc:1199] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a21ec49194e24723b31fd9c86f1f85e4
I20251023 20:15:32.825654 24780 ts_tablet_manager.cc:1950] T a21ec49194e24723b31fd9c86f1f85e4 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.827474 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 149) took 4061 ms. Trace:
I20251023 20:15:32.828496 24780 rpcz_store.cc:276] 1023 20:15:28.766035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.766179 (+ 144us) service_pool.cc:225] Handling call
1023 20:15:32.784699 (+4018520us) ts_tablet_manager.cc:1151] Deleting tablet a21ec49194e24723b31fd9c86f1f85e4
1023 20:15:32.784710 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.809968 (+ 25258us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.821436 (+ 11468us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.827449 (+ 6013us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4018354}
I20251023 20:15:32.829192 24780 tablet_replica.cc:333] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:32.829478 23787 tablet_bootstrap.cc:654] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.829664 24780 raft_consensus.cc:2243] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.830107 24780 raft_consensus.cc:2272] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.832679 24780 ts_tablet_manager.cc:1916] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.834483 23786 ts_tablet_manager.cc:1434] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.076s user 0.001s sys 0.001s
I20251023 20:15:32.825855 23790 consensus_queue.cc:260] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.837548 23790 ts_tablet_manager.cc:1434] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20251023 20:15:32.837903 23790 tablet_bootstrap.cc:492] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.839738 23790 tablet_bootstrap.cc:654] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.843194 23786 tablet_bootstrap.cc:492] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.849995 23786 tablet_bootstrap.cc:654] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.852833 25167 raft_consensus.cc:493] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.853044 25167 raft_consensus.cc:515] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.853732 25167 leader_election.cc:290] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.854451 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.854696 23668 raft_consensus.cc:2393] T 124703edbf08408a9ca69732674c0ad3 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:32.855075 23787 tablet_bootstrap.cc:492] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.855247 23787 ts_tablet_manager.cc:1403] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20251023 20:15:32.856034 23787 raft_consensus.cc:359] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.856220 23787 raft_consensus.cc:385] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.856294 23787 raft_consensus.cc:740] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.856568 23787 consensus_queue.cc:260] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.857215 23787 ts_tablet_manager.cc:1434] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:32.857509 23787 tablet_bootstrap.cc:492] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.859068 23787 tablet_bootstrap.cc:654] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.860503 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf (table default.Table_1fd972f3964245fab674e2cfc26a8210 [id=cf5e426caddf4508bc0780b9809fff5e]) successfully deleted
I20251023 20:15:32.864270 24780 ts_tablet_manager.cc:1929] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:32.864410 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b90810781f724c78935a33d868ffa27d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.864650 23528 raft_consensus.cc:2468] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:32.866236 23787 tablet_bootstrap.cc:492] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.866879 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a21ec49194e24723b31fd9c86f1f85e4 (table default.Table_fc18b4fbf71a45f6b3cfee31effd8eef [id=ebb0bfc95f844d50859f36a09ac81f4d]) successfully deleted
I20251023 20:15:32.867852 23787 ts_tablet_manager.cc:1403] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20251023 20:15:32.869055 23787 raft_consensus.cc:359] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.869566 23787 raft_consensus.cc:385] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.869680 23787 raft_consensus.cc:740] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.869935 23787 consensus_queue.cc:260] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.870682 23787 ts_tablet_manager.cc:1434] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:32.870990 23787 tablet_bootstrap.cc:492] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:32.871140 24780 log.cc:1199] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/40e0301fb5974ec5b6e8291230a736c2
I20251023 20:15:32.871565 24780 ts_tablet_manager.cc:1950] T 40e0301fb5974ec5b6e8291230a736c2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.873340 23787 tablet_bootstrap.cc:654] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.873739 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 150) took 4107 ms. Trace:
I20251023 20:15:32.881558 24780 rpcz_store.cc:276] 1023 20:15:28.766486 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.766631 (+ 145us) service_pool.cc:225] Handling call
1023 20:15:32.829141 (+4062510us) ts_tablet_manager.cc:1151] Deleting tablet 40e0301fb5974ec5b6e8291230a736c2
1023 20:15:32.829152 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.836445 (+ 7293us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.864124 (+ 27679us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.873718 (+ 9594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4062323}
I20251023 20:15:32.882143 24780 tablet_replica.cc:333] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:32.882823 24780 raft_consensus.cc:2243] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.883260 24780 raft_consensus.cc:2272] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.884047 24780 ts_tablet_manager.cc:1916] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.892026 23790 tablet_bootstrap.cc:492] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.892236 23790 ts_tablet_manager.cc:1403] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.004s
I20251023 20:15:32.893860 23786 tablet_bootstrap.cc:492] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:32.894464 23786 ts_tablet_manager.cc:1403] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.003s
I20251023 20:15:32.895733 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:32.897097 23790 raft_consensus.cc:359] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.897342 23790 raft_consensus.cc:385] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.897418 23790 raft_consensus.cc:740] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.897651 23790 consensus_queue.cc:260] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.898204 23790 ts_tablet_manager.cc:1434] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:32.898602 23790 tablet_bootstrap.cc:492] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:32.897987 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:32.901782 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "124703edbf08408a9ca69732674c0ad3" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.902144 23333 leader_election.cc:304] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.902385 25237 raft_consensus.cc:2804] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.902467 25237 raft_consensus.cc:493] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.902542 25237 raft_consensus.cc:3060] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.902566 23399 raft_consensus.cc:1275] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Refusing update from remote peer ea6853efbbcc484eb7529c7e9b37a47a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20251023 20:15:32.902658 23399 pending_rounds.cc:85] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48: Aborting all ops after (but not including) 0
I20251023 20:15:32.902712 23399 pending_rounds.cc:107] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251023 20:15:32.902806 23399 raft_consensus.cc:2889] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251023 20:15:32.903330 23333 leader_election.cc:343] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: b90810781f724c78935a33d868ffa27d
I20251023 20:15:32.901448 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:32.904119 24297 consensus_queue.cc:1048] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.904244 23786 raft_consensus.cc:359] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.905679 25237 raft_consensus.cc:515] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.906294 25237 leader_election.cc:290] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.906104 23786 raft_consensus.cc:385] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.906950 24780 ts_tablet_manager.cc:1929] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:32.906277 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:32.907094 24780 log.cc:1199] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/25079532547d4175802b84877790662d
I20251023 20:15:32.907368 24780 ts_tablet_manager.cc:1950] T 25079532547d4175802b84877790662d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:32.906632 23333 leader_election.cc:304] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:32.908089 24297 raft_consensus.cc:2804] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:32.908162 24297 raft_consensus.cc:493] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:32.908222 24297 raft_consensus.cc:3060] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.908283 25040 consensus_queue.cc:1048] T 19c58a406d564b048fb16a85aa0b754b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.907956 23786 raft_consensus.cc:740] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.909106 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:32.910683 24297 raft_consensus.cc:515] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.911252 24297 leader_election.cc:290] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.911672 25144 consensus_queue.cc:1048] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.912040 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 40e0301fb5974ec5b6e8291230a736c2 (table default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]) successfully deleted
I20251023 20:15:32.910948 23786 consensus_queue.cc:260] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.912443 23471 leader_election.cc:304] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:32.912837 25144 raft_consensus.cc:3060] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.911752 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:32.915402 25144 raft_consensus.cc:2749] T 124703edbf08408a9ca69732674c0ad3 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.916153 24830 consensus_queue.cc:1048] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.917255 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 151) took 4150 ms. Trace:
I20251023 20:15:32.917678 24830 consensus_queue.cc:1048] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.918198 23786 ts_tablet_manager.cc:1434] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20251023 20:15:32.920217 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:32.920450 23786 tablet_bootstrap.cc:492] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:32.921195 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:32.923405 25061 raft_consensus.cc:493] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.923537 25061 raft_consensus.cc:515] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.923355 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:32.924136 25061 leader_election.cc:290] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.924552 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5161336598884742ad403fc88c9d9f0b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.925220 23604 leader_election.cc:343] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.929106 25061 raft_consensus.cc:493] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.929250 25061 raft_consensus.cc:515] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.929855 25061 leader_election.cc:290] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.931113 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca11ddd5d97544cf916b87fd3dff219d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:32.931403 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d238e071cb4d49b173febd90bc909b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:32.931567 23666 raft_consensus.cc:3060] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:32.931802 23604 leader_election.cc:343] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.932171 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d238e071cb4d49b173febd90bc909b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.926162 23212 catalog_manager.cc:5649] T b5cb6b4e2e5e4a2f9183111cfd54ec90 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:32.932824 23531 raft_consensus.cc:3060] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.934209 23666 raft_consensus.cc:2468] T c3d238e071cb4d49b173febd90bc909b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.935530 23531 raft_consensus.cc:2468] T c3d238e071cb4d49b173febd90bc909b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.939581 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:32.942299 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:32.944702 25144 consensus_queue.cc:1048] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:32.924757 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5161336598884742ad403fc88c9d9f0b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.930713 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca11ddd5d97544cf916b87fd3dff219d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.945408 24780 rpcz_store.cc:276] 1023 20:15:28.766975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.767130 (+ 155us) service_pool.cc:225] Handling call
1023 20:15:32.882090 (+4114960us) ts_tablet_manager.cc:1151] Deleting tablet 25079532547d4175802b84877790662d
1023 20:15:32.882102 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.894547 (+ 12445us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.906742 (+ 12195us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:32.917232 (+ 10490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4114751}
I20251023 20:15:32.945746 24780 tablet_replica.cc:333] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
W20251023 20:15:32.945911 23605 leader_election.cc:343] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.946043 23605 leader_election.cc:304] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.946296 25250 raft_consensus.cc:2749] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:32.946638 23605 leader_election.cc:343] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.946739 23605 leader_election.cc:304] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.947409 25250 raft_consensus.cc:2749] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.951854 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b90810781f724c78935a33d868ffa27d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:32.949643 23790 tablet_bootstrap.cc:654] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.952052 23531 raft_consensus.cc:3060] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:32.954190 23786 tablet_bootstrap.cc:654] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:32.954743 23531 raft_consensus.cc:2468] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:32.959021 24780 raft_consensus.cc:2243] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:32.960999 24780 pending_rounds.cc:70] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:32.961129 24780 pending_rounds.cc:77] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082174416621568 op_type: NO_OP noop_request { }
I20251023 20:15:32.961323 24780 raft_consensus.cc:2889] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:32.957432 23212 catalog_manager.cc:5649] T c05b9bcea6ff49188758271401270227 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:32.963146 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 25079532547d4175802b84877790662d (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]) successfully deleted
I20251023 20:15:32.965497 24780 raft_consensus.cc:2272] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:32.966051 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:32.967365 24780 ts_tablet_manager.cc:1916] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:32.969375 25144 raft_consensus.cc:493] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.969532 25144 raft_consensus.cc:515] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.970163 25144 leader_election.cc:290] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:32.970852 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:32.968504 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:32.971139 23666 raft_consensus.cc:2393] T ac4d82d024fc491ab0dbe04147cb156c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:32.964305 23333 leader_election.cc:304] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:32.972280 24297 raft_consensus.cc:2804] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
W20251023 20:15:32.972327 23333 leader_election.cc:343] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: b90810781f724c78935a33d868ffa27d
I20251023 20:15:32.972407 24297 raft_consensus.cc:697] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:32.972750 24297 consensus_queue.cc:237] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.972734 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:32.974649 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac4d82d024fc491ab0dbe04147cb156c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.980623 25250 raft_consensus.cc:493] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:32.980795 25250 raft_consensus.cc:515] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.981523 25250 leader_election.cc:290] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:32.982074 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:32.982172 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:32.982700 23605 leader_election.cc:343] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:32.982853 23604 leader_election.cc:343] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:32.982960 23604 leader_election.cc:304] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:32.983336 25250 raft_consensus.cc:2749] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:32.987177 23787 tablet_bootstrap.cc:492] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:32.987495 23787 ts_tablet_manager.cc:1403] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.117s user 0.006s sys 0.001s
I20251023 20:15:32.988152 23790 tablet_bootstrap.cc:492] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:32.988332 23790 ts_tablet_manager.cc:1403] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.008s
I20251023 20:15:32.988983 23787 raft_consensus.cc:359] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.989138 23790 raft_consensus.cc:359] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.989291 23790 raft_consensus.cc:385] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.989347 23790 raft_consensus.cc:740] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.989326 23787 raft_consensus.cc:385] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:32.989530 23790 consensus_queue.cc:260] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.989598 23787 raft_consensus.cc:740] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:32.989998 23790 ts_tablet_manager.cc:1434] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:32.990134 23787 consensus_queue.cc:260] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:32.992444 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:32.994220 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:32.994199 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:32.998118 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:32.999951 23787 ts_tablet_manager.cc:1434] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20251023 20:15:33.001915 23787 tablet_bootstrap.cc:492] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.003525 24297 consensus_queue.cc:1048] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.004359 23790 tablet_bootstrap.cc:492] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:33.005687 23333 leader_election.cc:304] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:33.005971 24297 raft_consensus.cc:2804] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.006080 24297 raft_consensus.cc:697] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:33.005364 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:33.006310 24297 consensus_queue.cc:237] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.010474 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:33.012337 23786 tablet_bootstrap.cc:492] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.012593 23786 ts_tablet_manager.cc:1403] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.000s
I20251023 20:15:33.013506 23786 raft_consensus.cc:359] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.013873 23786 raft_consensus.cc:385] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.014038 23786 raft_consensus.cc:740] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.014479 23786 consensus_queue.cc:260] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.016345 25250 raft_consensus.cc:493] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.016479 25250 raft_consensus.cc:515] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.017122 25250 leader_election.cc:290] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.019080 23790 tablet_bootstrap.cc:654] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.020740 23787 tablet_bootstrap.cc:654] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.022081 23786 ts_tablet_manager.cc:1434] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:33.023190 25250 raft_consensus.cc:493] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.023327 25250 raft_consensus.cc:515] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.023945 25250 leader_election.cc:290] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.024502 23471 leader_election.cc:304] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:33.024699 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76380b341a304d6ba31648fbd1548f05" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.024919 25144 raft_consensus.cc:3060] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:33.025252 23604 leader_election.cc:343] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.026877 25167 raft_consensus.cc:493] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.027014 25167 raft_consensus.cc:515] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.027578 25167 leader_election.cc:290] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.028158 25250 raft_consensus.cc:493] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.028288 25250 raft_consensus.cc:515] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.028942 25250 leader_election.cc:290] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.029372 23786 tablet_bootstrap.cc:492] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.029609 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e658aa360e524a9f9bdb35c1f68ae426" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.030689 23604 leader_election.cc:343] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.030897 25144 raft_consensus.cc:2749] T ac4d82d024fc491ab0dbe04147cb156c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.031025 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6480cdc1afda4671a346044f232b083e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.031194 23669 raft_consensus.cc:2468] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:33.031570 23470 leader_election.cc:304] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.030800 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:33.032181 25167 raft_consensus.cc:2804] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.032315 25167 raft_consensus.cc:493] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.032394 25167 raft_consensus.cc:3060] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.020720 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53f33020319d48c791e1156c67b55be6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.030133 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e658aa360e524a9f9bdb35c1f68ae426" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.030316 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76380b341a304d6ba31648fbd1548f05" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.035012 25167 raft_consensus.cc:515] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.036420 25237 raft_consensus.cc:493] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:33.038168 23605 leader_election.cc:343] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:33.038364 23605 leader_election.cc:343] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.038456 23605 leader_election.cc:304] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:33.038822 23605 leader_election.cc:343] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.038903 23605 leader_election.cc:304] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.039273 25061 raft_consensus.cc:493] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.039384 25061 raft_consensus.cc:515] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.039958 25061 leader_election.cc:290] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.040083 25061 raft_consensus.cc:2749] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.040211 25061 raft_consensus.cc:2749] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.041181 25237 raft_consensus.cc:515] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.041723 25237 leader_election.cc:290] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.042037 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.043082 23604 leader_election.cc:343] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.045189 23786 tablet_bootstrap.cc:654] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.045300 24780 ts_tablet_manager.cc:1929] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:33.045404 24780 log.cc:1199] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/ac55e14745534a02a2f6a422cab6b149
I20251023 20:15:33.045665 25167 leader_election.cc:290] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.045724 24780 ts_tablet_manager.cc:1950] T ac55e14745534a02a2f6a422cab6b149 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.046039 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 152) took 4278 ms. Trace:
I20251023 20:15:33.046125 24780 rpcz_store.cc:276] 1023 20:15:28.767441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.767579 (+ 138us) service_pool.cc:225] Handling call
1023 20:15:32.945691 (+4178112us) ts_tablet_manager.cc:1151] Deleting tablet ac55e14745534a02a2f6a422cab6b149
1023 20:15:32.945703 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:32.970480 (+ 24777us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.045179 (+ 74699us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.046020 (+ 841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4177915}
I20251023 20:15:33.046423 24780 tablet_replica.cc:333] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.046608 24780 raft_consensus.cc:2243] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.046979 24780 raft_consensus.cc:2272] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.047538 24780 ts_tablet_manager.cc:1916] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.056063 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:33.056456 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:33.056753 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:33.056985 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:33.057623 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:33.058205 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:33.058732 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:33.059077 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:33.059689 25250 raft_consensus.cc:493] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.059860 25250 raft_consensus.cc:515] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.061036 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d62dbf044acd4459b26d2e7e50f6206e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.061684 23604 leader_election.cc:343] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.068961 23534 raft_consensus.cc:3060] T e5af107f9a144edb9080b7135ab8aeb9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.072604 25061 consensus_queue.cc:1048] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.073239 24780 ts_tablet_manager.cc:1929] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:33.073359 24780 log.cc:1199] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/fe922c438a894a7aa04699b2d2897a28
I20251023 20:15:33.073679 24780 ts_tablet_manager.cc:1950] T fe922c438a894a7aa04699b2d2897a28 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.060490 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:33.081836 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:33.082592 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 154) took 4314 ms. Trace:
I20251023 20:15:33.082532 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:33.082911 24780 rpcz_store.cc:276] 1023 20:15:28.768389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.768549 (+ 160us) service_pool.cc:225] Handling call
1023 20:15:33.046379 (+4277830us) ts_tablet_manager.cc:1151] Deleting tablet fe922c438a894a7aa04699b2d2897a28
1023 20:15:33.046389 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.065168 (+ 18779us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.073120 (+ 7952us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.082566 (+ 9446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4277604}
I20251023 20:15:33.083621 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 156) took 4314 ms. Trace:
I20251023 20:15:33.084945 24780 rpcz_store.cc:276] 1023 20:15:28.769304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.769458 (+ 154us) service_pool.cc:225] Handling call
1023 20:15:33.083473 (+4314015us) ts_tablet_manager.cc:1151] Deleting tablet 3befdfc85bca42bb816e11d1dd832c8c
1023 20:15:33.083487 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.083601 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4313848}
I20251023 20:15:33.085661 24780 tablet_replica.cc:333] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.086306 24780 raft_consensus.cc:2243] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.086616 24780 raft_consensus.cc:2272] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.087610 24780 ts_tablet_manager.cc:1916] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.088092 25250 leader_election.cc:290] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:33.088460 23605 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 49642.6us
I20251023 20:15:33.089318 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6480cdc1afda4671a346044f232b083e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.089494 23669 raft_consensus.cc:3060] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.090108 25144 consensus_queue.cc:1048] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.091938 23669 raft_consensus.cc:2468] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:33.095774 23786 tablet_bootstrap.cc:492] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.096088 23786 ts_tablet_manager.cc:1403] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.000s
I20251023 20:15:33.096851 23470 leader_election.cc:304] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.097134 23786 raft_consensus.cc:359] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.097223 25144 raft_consensus.cc:2804] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.097489 25144 raft_consensus.cc:697] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:33.097836 25144 consensus_queue.cc:237] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.100392 23786 raft_consensus.cc:385] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.100711 23786 raft_consensus.cc:740] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.089020 23790 tablet_bootstrap.cc:492] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:33.089397 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.101473 23790 ts_tablet_manager.cc:1403] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.006s
I20251023 20:15:33.092780 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d62dbf044acd4459b26d2e7e50f6206e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:33.102162 23605 leader_election.cc:343] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.102330 23605 leader_election.cc:304] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.102358 23790 raft_consensus.cc:359] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:33.102742 23605 leader_election.cc:343] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.102762 25250 raft_consensus.cc:2749] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.102903 23605 leader_election.cc:304] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.103097 25250 raft_consensus.cc:2749] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.103510 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53f33020319d48c791e1156c67b55be6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.103698 23790 raft_consensus.cc:385] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.103775 23790 raft_consensus.cc:740] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:33.104034 23604 leader_election.cc:343] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.104130 23604 leader_election.cc:304] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.102511 23786 consensus_queue.cc:260] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.104393 23790 consensus_queue.cc:260] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.104499 25264 raft_consensus.cc:2749] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.105156 23790 ts_tablet_manager.cc:1434] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:33.106575 23790 tablet_bootstrap.cc:492] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:33.107072 25237 consensus_queue.cc:1048] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.108741 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.109036 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.109221 23666 raft_consensus.cc:2468] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.110198 23532 raft_consensus.cc:3060] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.114657 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet ac55e14745534a02a2f6a422cab6b149 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]) successfully deleted
I20251023 20:15:33.116084 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet fe922c438a894a7aa04699b2d2897a28 (table default.Table_aab23aa8d50a4902b09495b3fa1ea6ca [id=fe2df9a958ec4863ab2100750ba7dc5e]) successfully deleted
W20251023 20:15:33.116384 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet 3befdfc85bca42bb816e11d1dd832c8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3befdfc85bca42bb816e11d1dd832c8c already in progress: creating tablet
I20251023 20:15:33.117288 23790 tablet_bootstrap.cc:654] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.117460 23786 ts_tablet_manager.cc:1434] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20251023 20:15:33.117913 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:33.120453 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:33.121024 25250 raft_consensus.cc:493] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.121169 25250 raft_consensus.cc:515] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.121753 25250 leader_election.cc:290] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.122109 23786 tablet_bootstrap.cc:492] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.122483 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dae906f257c346d7a1b28c9052d600b5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.122983 23604 leader_election.cc:343] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.124336 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:33.130630 23786 tablet_bootstrap.cc:654] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.133484 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:33.134671 24780 ts_tablet_manager.cc:1929] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:33.135062 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_6621b13baee14821a56e8610ed7c68a7 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 1 got 0
I20251023 20:15:33.138548 24780 log.cc:1199] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/58ae6a6e58274b8baf952eaaefde7127
I20251023 20:15:33.140564 24780 ts_tablet_manager.cc:1950] T 58ae6a6e58274b8baf952eaaefde7127 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.141422 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 157) took 4371 ms. Trace:
I20251023 20:15:33.141657 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dae906f257c346d7a1b28c9052d600b5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:33.142411 23605 leader_election.cc:343] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.142581 23605 leader_election.cc:304] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.143030 25264 raft_consensus.cc:2749] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.143281 24780 rpcz_store.cc:276] 1023 20:15:28.769790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.769958 (+ 168us) service_pool.cc:225] Handling call
1023 20:15:33.085610 (+4315652us) ts_tablet_manager.cc:1151] Deleting tablet 58ae6a6e58274b8baf952eaaefde7127
1023 20:15:33.085623 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.111166 (+ 25543us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.134496 (+ 23330us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.141401 (+ 6905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4315466}
I20251023 20:15:33.143890 24780 tablet_replica.cc:333] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.144315 24780 raft_consensus.cc:2243] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.145421 24780 raft_consensus.cc:2272] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.147416 24780 ts_tablet_manager.cc:1916] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.146613 23212 catalog_manager.cc:5649] T 165d62dd42da460faccec706f08272a1 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:33.153409 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:33.156334 25250 raft_consensus.cc:493] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.156490 25250 raft_consensus.cc:515] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.156596 25061 consensus_queue.cc:1048] T e5af107f9a144edb9080b7135ab8aeb9 P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.157066 25250 leader_election.cc:290] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.157639 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e36bb5e61d34b35a373bd6a51a7ac5e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.158164 23604 leader_election.cc:343] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.158062 23212 catalog_manager.cc:5649] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:33.159588 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:33.164817 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e36bb5e61d34b35a373bd6a51a7ac5e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:33.165652 23605 leader_election.cc:343] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.166219 23605 leader_election.cc:304] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.167565 25264 raft_consensus.cc:2749] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.167284 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:33.173709 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6480cdc1afda4671a346044f232b083e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.173987 23399 raft_consensus.cc:2468] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:33.174741 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6480cdc1afda4671a346044f232b083e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:33.174896 23396 raft_consensus.cc:3060] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.176306 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 58ae6a6e58274b8baf952eaaefde7127 (table default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]) successfully deleted
I20251023 20:15:33.176658 25235 raft_consensus.cc:493] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:33.176697 23333 leader_election.cc:343] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.176780 25235 raft_consensus.cc:515] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.177384 25040 consensus_queue.cc:1048] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.177979 25172 consensus_queue.cc:1048] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.181015 25235 leader_election.cc:290] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.191428 23786 tablet_bootstrap.cc:492] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.194977 23786 ts_tablet_manager.cc:1403] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.002s
I20251023 20:15:33.195989 23786 raft_consensus.cc:359] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.199457 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:33.200417 23786 raft_consensus.cc:385] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.200621 23786 raft_consensus.cc:740] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.200960 23786 consensus_queue.cc:260] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.202059 23786 ts_tablet_manager.cc:1434] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251023 20:15:33.202239 25276 raft_consensus.cc:493] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.202610 23786 tablet_bootstrap.cc:492] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.204372 23786 tablet_bootstrap.cc:654] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.204527 25276 raft_consensus.cc:515] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.203109 23212 catalog_manager.cc:5649] T 98a4553c1f014ef3ac975bf7a8076334 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:33.205747 25276 leader_election.cc:290] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.206239 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01fa5f9c700b458ea147373d78551573" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.207984 23604 leader_election.cc:343] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.207618 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:33.211349 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
W20251023 20:15:33.213641 23213 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc488f0beec14d0eb869b7474ec16205 consensus state has no leader
I20251023 20:15:33.213867 23213 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 67) took 6551 ms. Trace:
I20251023 20:15:33.213972 23213 rpcz_store.cc:276] 1023 20:15:26.661886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.001291 (+4339405us) service_pool.cc:225] Handling call
1023 20:15:31.001683 (+ 392us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:31.001932 (+ 249us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:31.002120 (+ 188us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:31.002174 (+ 54us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:31.018513 (+ 16339us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:33.213798 (+2195285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.002756 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.002858 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.002865 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:31.002867 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:31.002986 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.002998 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.003002 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:31.003057 (+ 55us) tablet.cc:699] Row locks acquired
1023 20:15:31.003059 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:31.003092 (+ 33us) write_op.cc:270] Start()
1023 20:15:31.003120 (+ 28us) write_op.cc:276] Timestamp: P: 1761250531003050 usec, L: 0
1023 20:15:31.003123 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.003285 (+ 162us) log.cc:844] Serialized 405 byte log entry
1023 20:15:31.011622 (+ 8337us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.011733 (+ 111us) write_op.cc:301] APPLY: starting
1023 20:15:31.011750 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.011794 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.011832 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.011896 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.011928 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:31.011935 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:31.012030 (+ 95us) log.cc:844] Serialized 23 byte log entry
1023 20:15:31.012097 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.012116 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:31.012124 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.012197 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.012201 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"apply.run_cpu_time_us":453,"apply.run_wall_time_us":452,"num_ops":1,"prepare.queue_time_us":68,"prepare.run_cpu_time_us":623,"prepare.run_wall_time_us":624,"raft.queue_time_us":699,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":216,"replication_time_us":8466}]]}
W20251023 20:15:33.214733 23213 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 169) took 1301 ms (client timeout 1000 ms). Trace:
W20251023 20:15:33.214826 23213 rpcz_store.cc:269] 1023 20:15:31.913423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.214648 (+1301225us) service_pool.cc:212] Skipping call since client already timed out
1023 20:15:33.214687 (+ 39us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20251023 20:15:33.215171 23213 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 195) took 1223 ms (client timeout 1000 ms). Trace:
W20251023 20:15:33.215256 23213 rpcz_store.cc:269] 1023 20:15:31.991717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.215097 (+1223380us) service_pool.cc:212] Skipping call since client already timed out
1023 20:15:33.215126 (+ 29us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 20:15:33.217207 23210 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet ea3149f906064f8c8167292ec86540fd
I20251023 20:15:33.217727 23210 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 76) took 5649 ms. Trace:
I20251023 20:15:33.217835 23210 rpcz_store.cc:276] 1023 20:15:27.568688 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.286909 (+3718221us) service_pool.cc:225] Handling call
1023 20:15:31.287374 (+ 465us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:31.287513 (+ 139us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:31.287575 (+ 62us) catalog_manager.cc:2824] Locking tablets
1023 20:15:31.423141 (+135566us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:31.433011 (+ 9870us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:31.509888 (+ 76877us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:33.217157 (+1707269us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:33.217663 (+ 506us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.423992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.424115 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.424123 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:31.424126 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:31.424265 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.424280 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.424286 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.424409 (+ 123us) tablet.cc:699] Row locks acquired
1023 20:15:31.424411 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:31.424451 (+ 40us) write_op.cc:270] Start()
1023 20:15:31.424500 (+ 49us) write_op.cc:276] Timestamp: P: 1761250531424410 usec, L: 0
1023 20:15:31.424504 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.424711 (+ 207us) log.cc:844] Serialized 734 byte log entry
1023 20:15:31.430139 (+ 5428us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.430611 (+ 472us) write_op.cc:301] APPLY: starting
1023 20:15:31.430628 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.430677 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.430713 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.430781 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.430816 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:31.430823 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:31.430911 (+ 88us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.430972 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.430992 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:31.430999 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.431067 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.431070 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":407,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":454,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":832,"raft.queue_time_us":76,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":289,"replication_time_us":5600}]]}
I20251023 20:15:33.223232 23333 leader_election.cc:304] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.223915 24297 raft_consensus.cc:2804] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.223999 24297 raft_consensus.cc:493] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.224069 24297 raft_consensus.cc:3060] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.225222 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01fa5f9c700b458ea147373d78551573" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.225939 23787 tablet_bootstrap.cc:492] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.226089 23787 ts_tablet_manager.cc:1403] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.225s user 0.022s sys 0.051s
I20251023 20:15:33.230646 24830 consensus_queue.cc:1048] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.230839 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet ea3149f906064f8c8167292ec86540fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.231149 24830 consensus_queue.cc:1048] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.231338 23219 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 26a680ad3aed487593bd1b63004dbbfc
I20251023 20:15:33.231880 23219 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 66) took 6572 ms. Trace:
I20251023 20:15:33.232697 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet ea3149f906064f8c8167292ec86540fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:33.233676 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet 26a680ad3aed487593bd1b63004dbbfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:30 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.235373 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 26a680ad3aed487593bd1b63004dbbfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:30 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:33.236651 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.236893 23669 raft_consensus.cc:2468] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:33.238586 23470 consensus_peers.cc:597] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a86a25659ec44d7c825783036342ed45. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:33.239996 23787 raft_consensus.cc:359] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.241289 25040 raft_consensus.cc:493] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.241496 25040 raft_consensus.cc:515] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.242148 25040 leader_election.cc:290] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.242717 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet ea3149f906064f8c8167292ec86540fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:33.242689 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.242879 23787 raft_consensus.cc:385] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.242960 23787 raft_consensus.cc:740] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.243049 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 26a680ad3aed487593bd1b63004dbbfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:30 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:33.243784 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:33.247040 23787 consensus_queue.cc:260] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.249289 25236 raft_consensus.cc:493] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.249173 25264 raft_consensus.cc:493] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.249454 25236 raft_consensus.cc:515] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.249537 25264 raft_consensus.cc:515] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.250133 25264 leader_election.cc:290] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:33.250573 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6f3abcaa81c4646bc1432bcbd1d4fcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.250759 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6f3abcaa81c4646bc1432bcbd1d4fcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.251327 23604 leader_election.cc:343] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.253675 23787 ts_tablet_manager.cc:1434] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20251023 20:15:33.254976 25265 consensus_queue.cc:1048] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:33.256520 23470 consensus_peers.cc:597] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb01b3f4ff7f41e98063ebf76abeace6. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:33.257107 25144 raft_consensus.cc:493] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d220aba7c9c0400398ffde917376ee48)
I20251023 20:15:33.257328 23787 tablet_bootstrap.cc:492] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.257302 25144 raft_consensus.cc:515] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.258114 25144 leader_election.cc:290] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.259155 23333 leader_election.cc:304] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.259168 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.259775 25237 raft_consensus.cc:2804] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.259889 25237 raft_consensus.cc:493] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.259992 25237 raft_consensus.cc:3060] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:33.262105 23333 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:33.261598 23219 rpcz_store.cc:276] 1023 20:15:26.658954 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.842118 (+4183164us) service_pool.cc:225] Handling call
1023 20:15:30.907779 (+ 65661us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:30.915293 (+ 7514us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:30.915360 (+ 67us) catalog_manager.cc:2824] Locking tablets
1023 20:15:31.367790 (+452430us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:31.377305 (+ 9515us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:31.377398 (+ 93us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:33.224054 (+1846656us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:33.231824 (+ 7770us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.368856 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.368976 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.368983 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:31.368985 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:31.369122 (+ 137us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.369135 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.369140 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:31.369246 (+ 106us) tablet.cc:699] Row locks acquired
1023 20:15:31.369248 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:31.369289 (+ 41us) write_op.cc:270] Start()
1023 20:15:31.369321 (+ 32us) write_op.cc:276] Timestamp: P: 1761250531369250 usec, L: 0
1023 20:15:31.369326 (+ 5us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.369511 (+ 185us) log.cc:844] Serialized 765 byte log entry
1023 20:15:31.373378 (+ 3867us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.376404 (+ 3026us) write_op.cc:301] APPLY: starting
1023 20:15:31.376423 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.376467 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.376502 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.376662 (+ 160us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.376697 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:31.376704 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:31.376817 (+ 113us) log.cc:844] Serialized 29 byte log entry
1023 20:15:31.377194 (+ 377us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.377214 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:31.377221 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:31.377466 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.377471 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2945,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":1057,"num_ops":2,"prepare.queue_time_us":238,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":770,"raft.queue_time_us":581,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":253,"replication_time_us":4022}]]}
I20251023 20:15:33.264786 23787 tablet_bootstrap.cc:654] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:33.271224 23333 leader_election.cc:343] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.280514 23787 tablet_bootstrap.cc:492] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.283217 23787 ts_tablet_manager.cc:1403] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20251023 20:15:33.283149 25144 consensus_queue.cc:1048] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.285393 23787 raft_consensus.cc:359] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.285573 23787 raft_consensus.cc:385] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.285635 23787 raft_consensus.cc:740] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.286954 23394 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5407a850ec7045158f53cb8abf6fcb58" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.309903 23471 leader_election.cc:304] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:33.310525 23787 consensus_queue.cc:260] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.312351 24821 raft_consensus.cc:493] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.312531 24821 raft_consensus.cc:515] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.313246 24821 leader_election.cc:290] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.314055 23787 ts_tablet_manager.cc:1434] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.029s user 0.001s sys 0.000s
W20251023 20:15:33.314705 23333 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:33.315339 23333 leader_election.cc:304] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.315747 23787 tablet_bootstrap.cc:492] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.316150 25142 raft_consensus.cc:2749] T 5407a850ec7045158f53cb8abf6fcb58 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:33.316404 24821 raft_consensus.cc:2749] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.321651 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.322723 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.322916 23669 raft_consensus.cc:2468] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:33.323053 23949 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.127.189:33509 timed out after 0.621s (SENT)
I20251023 20:15:33.323987 24821 consensus_queue.cc:1048] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.324602 23333 leader_election.cc:304] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.325814 23787 tablet_bootstrap.cc:654] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.326406 24821 raft_consensus.cc:2804] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.326522 24821 raft_consensus.cc:493] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.326604 24821 raft_consensus.cc:3060] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.330871 24821 raft_consensus.cc:515] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.334031 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
W20251023 20:15:33.334864 23333 leader_election.cc:343] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:33.336094 23333 leader_election.cc:343] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.331616 25264 raft_consensus.cc:493] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.336516 25264 raft_consensus.cc:515] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.337126 25264 leader_election.cc:290] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.337422 23786 tablet_bootstrap.cc:492] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.337643 23786 ts_tablet_manager.cc:1403] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.006s
I20251023 20:15:33.338081 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a744214a24cc4bf28654ffa417ce0d3d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.338586 23604 leader_election.cc:343] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.338562 23786 raft_consensus.cc:359] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.338727 23786 raft_consensus.cc:385] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.338948 23786 raft_consensus.cc:740] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.338197 23394 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a744214a24cc4bf28654ffa417ce0d3d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:33.340601 23605 leader_election.cc:343] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.340709 23605 leader_election.cc:304] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.341125 25264 raft_consensus.cc:2749] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.346649 25235 consensus_queue.cc:1048] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:33.350986 23605 leader_election.cc:343] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.351120 23605 leader_election.cc:304] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.351508 25264 raft_consensus.cc:2749] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:33.351925 23605 leader_election.cc:343] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.352022 23605 leader_election.cc:304] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.352388 23787 tablet_bootstrap.cc:492] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.352422 25264 raft_consensus.cc:2749] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.352890 25236 leader_election.cc:290] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.352969 23787 ts_tablet_manager.cc:1403] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20251023 20:15:33.355314 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92c50c4845524163834861438291a3ec" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.356043 23790 tablet_bootstrap.cc:492] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:33.356128 24821 leader_election.cc:290] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.356228 23790 ts_tablet_manager.cc:1403] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.250s user 0.005s sys 0.000s
I20251023 20:15:33.356727 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92c50c4845524163834861438291a3ec" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.356913 23669 raft_consensus.cc:2468] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.359045 23396 raft_consensus.cc:2468] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:33.361622 25237 raft_consensus.cc:515] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.362260 25237 leader_election.cc:290] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.361653 23787 raft_consensus.cc:359] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.363070 23787 raft_consensus.cc:385] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.363797 23787 raft_consensus.cc:740] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.365219 24297 raft_consensus.cc:515] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.365834 24297 leader_election.cc:290] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.365836 23787 consensus_queue.cc:260] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.370601 24780 ts_tablet_manager.cc:1929] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:33.370743 24780 log.cc:1199] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a27f192792cf41288d69ca15ab538472
I20251023 20:15:33.371143 24780 ts_tablet_manager.cc:1950] T a27f192792cf41288d69ca15ab538472 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.371299 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.371476 23668 raft_consensus.cc:3060] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.371491 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 158) took 4601 ms. Trace:
I20251023 20:15:33.371600 24780 rpcz_store.cc:276] 1023 20:15:28.770316 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.770493 (+ 177us) service_pool.cc:225] Handling call
1023 20:15:33.143838 (+4373345us) ts_tablet_manager.cc:1151] Deleting tablet a27f192792cf41288d69ca15ab538472
1023 20:15:33.143850 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.361469 (+217619us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.370439 (+ 8970us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.371459 (+ 1020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4373181}
I20251023 20:15:33.371933 24780 tablet_replica.cc:333] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.371861 23787 ts_tablet_manager.cc:1434] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20251023 20:15:33.372140 24780 raft_consensus.cc:2243] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.372779 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:33.373605 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:33.373929 23668 raft_consensus.cc:2468] T a2d4ba34cc444baf9fe725bc81478517 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:33.374275 23787 tablet_bootstrap.cc:492] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.374585 23790 raft_consensus.cc:359] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.374787 23790 raft_consensus.cc:385] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.374809 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.374881 23790 raft_consensus.cc:740] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.374943 23669 raft_consensus.cc:3060] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.375103 23790 consensus_queue.cc:260] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.375578 23790 ts_tablet_manager.cc:1434] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251023 20:15:33.375892 23790 tablet_bootstrap.cc:492] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:33.377888 23669 raft_consensus.cc:2468] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:33.378206 23790 tablet_bootstrap.cc:654] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.379124 25142 consensus_queue.cc:1048] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.379369 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.379745 23668 raft_consensus.cc:3060] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.386727 23333 leader_election.cc:304] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.386665 23787 tablet_bootstrap.cc:654] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.387389 24297 raft_consensus.cc:2804] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.387482 24297 raft_consensus.cc:493] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.387552 24297 raft_consensus.cc:3060] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.388545 24780 pending_rounds.cc:70] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:33.388810 24780 pending_rounds.cc:77] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082183196471296 op_type: NO_OP noop_request { }
I20251023 20:15:33.389328 24780 raft_consensus.cc:2889] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:33.389539 23668 raft_consensus.cc:2468] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:33.389750 24780 raft_consensus.cc:2272] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.390657 24297 raft_consensus.cc:515] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.391386 24297 leader_election.cc:290] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.391788 24780 ts_tablet_manager.cc:1916] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.392844 23333 leader_election.cc:304] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.393517 25237 raft_consensus.cc:2804] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.393625 25237 raft_consensus.cc:697] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:33.393891 25237 consensus_queue.cc:237] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:33.404549 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet ea3149f906064f8c8167292ec86540fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea3149f906064f8c8167292ec86540fd already in progress: creating tablet
W20251023 20:15:33.406728 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ea3149f906064f8c8167292ec86540fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea3149f906064f8c8167292ec86540fd already in progress: creating tablet
I20251023 20:15:33.407083 25282 tablet_replica.cc:333] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:33.407083 23333 leader_election.cc:304] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.407361 25282 raft_consensus.cc:2243] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.407555 25282 raft_consensus.cc:2272] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.409495 24297 raft_consensus.cc:2804] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.409729 24297 raft_consensus.cc:697] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:33.410076 24297 consensus_queue.cc:237] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.411183 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:33.414763 23333 leader_election.cc:304] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.416653 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:33.416779 24821 raft_consensus.cc:2804] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.416934 24821 raft_consensus.cc:697] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:33.417202 24821 consensus_queue.cc:237] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.422912 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a27f192792cf41288d69ca15ab538472 (table default.Table_bd80c8158a014c8192a0800e8c1c32b6 [id=d199b1ef8bfe4b94963604bfffe6f2e4]) successfully deleted
W20251023 20:15:33.426211 23204 catalog_manager.cc:4965] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): delete failed for tablet 26a680ad3aed487593bd1b63004dbbfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26a680ad3aed487593bd1b63004dbbfc already in progress: creating tablet
I20251023 20:15:33.432427 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92c50c4845524163834861438291a3ec" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:33.435859 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92c50c4845524163834861438291a3ec" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.436101 23668 raft_consensus.cc:3060] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:33.436967 23333 leader_election.cc:343] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:33.438213 23333 leader_election.cc:343] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.438733 23668 raft_consensus.cc:2468] T 92c50c4845524163834861438291a3ec P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:33.439698 25282 ts_tablet_manager.cc:1916] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.440704 25236 consensus_queue.cc:1048] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:33.441306 23333 consensus_peers.cc:597] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:33.441550 25172 consensus_queue.cc:1048] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.439213 23790 tablet_bootstrap.cc:492] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:33.448089 23790 ts_tablet_manager.cc:1403] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.005s
I20251023 20:15:33.450471 23333 leader_election.cc:304] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.453008 25236 raft_consensus.cc:2804] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.453150 25236 raft_consensus.cc:697] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:33.453425 25236 consensus_queue.cc:237] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.457614 24780 ts_tablet_manager.cc:1929] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:33.458449 24780 log.cc:1199] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/b90810781f724c78935a33d868ffa27d
I20251023 20:15:33.451596 23790 raft_consensus.cc:359] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.461956 23790 raft_consensus.cc:385] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.465833 23787 tablet_bootstrap.cc:492] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.466115 23787 ts_tablet_manager.cc:1403] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.001s
I20251023 20:15:33.476262 23790 raft_consensus.cc:740] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.477321 23790 consensus_queue.cc:260] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.462615 24780 ts_tablet_manager.cc:1950] T b90810781f724c78935a33d868ffa27d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.480171 24821 consensus_queue.cc:1048] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.480808 23790 ts_tablet_manager.cc:1434] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20251023 20:15:33.485999 23787 raft_consensus.cc:359] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.486785 23787 raft_consensus.cc:385] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.487129 23787 raft_consensus.cc:740] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.488931 23787 consensus_queue.cc:260] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.480970 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 159) took 4710 ms. Trace:
I20251023 20:15:33.481851 23446 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:33.492637 24780 rpcz_store.cc:276] 1023 20:15:28.770844 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.771298 (+ 454us) service_pool.cc:225] Handling call
1023 20:15:33.371859 (+4600561us) ts_tablet_manager.cc:1151] Deleting tablet b90810781f724c78935a33d868ffa27d
1023 20:15:33.371879 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.413317 (+ 41438us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.457006 (+ 43689us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.480952 (+ 23946us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16455,"tablet-delete.queue_time_us":4600382}
I20251023 20:15:33.493021 24780 tablet_replica.cc:333] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.493247 24780 raft_consensus.cc:2243] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.493613 24780 raft_consensus.cc:2272] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.493661 23787 ts_tablet_manager.cc:1434] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20251023 20:15:33.494189 24780 ts_tablet_manager.cc:1916] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.496061 23787 tablet_bootstrap.cc:492] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.500344 24780 ts_tablet_manager.cc:1929] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:33.500490 24780 log.cc:1199] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/15ced74d0dd74e5195ed98358957d0dd
I20251023 20:15:33.500898 24780 ts_tablet_manager.cc:1950] T 15ced74d0dd74e5195ed98358957d0dd P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.501600 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 160) took 4729 ms. Trace:
I20251023 20:15:33.501694 24780 rpcz_store.cc:276] 1023 20:15:28.771641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.771884 (+ 243us) service_pool.cc:225] Handling call
1023 20:15:33.492968 (+4721084us) ts_tablet_manager.cc:1151] Deleting tablet 15ced74d0dd74e5195ed98358957d0dd
1023 20:15:33.492979 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.497349 (+ 4370us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.500215 (+ 2866us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.501580 (+ 1365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4720936}
I20251023 20:15:33.502027 24780 tablet_replica.cc:333] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.502221 24780 raft_consensus.cc:2243] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.502573 24780 raft_consensus.cc:2272] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.502852 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.503154 24780 ts_tablet_manager.cc:1916] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.504133 25282 ts_tablet_manager.cc:1929] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:33.504894 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.505749 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.505946 25282 log.cc:1199] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/26a680ad3aed487593bd1b63004dbbfc
I20251023 20:15:33.506283 25282 ts_tablet_manager.cc:1950] T 26a680ad3aed487593bd1b63004dbbfc P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:33.339260 23786 consensus_queue.cc:260] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.507242 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.507726 23790 tablet_bootstrap.cc:492] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:33.507896 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc]) successfully deleted
I20251023 20:15:33.508728 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.509502 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet b90810781f724c78935a33d868ffa27d (table default.Table_fab9daaa8e644d2a9681e2229696c2bb [id=2df3ca33fcf54a978b31fc24b9d21ced]) successfully deleted
I20251023 20:15:33.509759 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 15ced74d0dd74e5195ed98358957d0dd (table default.Table_e62c8689605f4db7b6e5016d5dcda175 [id=91e8d69e8f264c6dafd1173e05f7be06]) successfully deleted
I20251023 20:15:33.510136 23531 raft_consensus.cc:2468] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.511654 23786 ts_tablet_manager.cc:1434] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.174s user 0.009s sys 0.000s
I20251023 20:15:33.512413 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.513038 25282 tablet_replica.cc:333] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:33.513685 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.514012 25282 raft_consensus.cc:2243] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.516168 23786 tablet_bootstrap.cc:492] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.516781 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet d2ae19cc80194363b8a9c96fbafb4ebe with delete_type TABLET_DATA_DELETED (replaced by tablet 94631fca1b834bc89791cb07fdc93523) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.517066 25282 raft_consensus.cc:2272] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.517673 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.519312 25282 ts_tablet_manager.cc:1916] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.521714 23790 tablet_bootstrap.cc:654] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.523743 23786 tablet_bootstrap.cc:654] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.520689 23650 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.531213 23642 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.525447 23632 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.525588 23633 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.525858 23631 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.525998 23649 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.526176 23634 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.526633 23636 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.526835 23647 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.527097 23641 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.527297 23645 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.527680 23638 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.528936 23640 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.529435 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.530426 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.536672 25236 consensus_queue.cc:1048] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.538014 25264 raft_consensus.cc:493] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.538187 25264 raft_consensus.cc:515] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.538782 25264 leader_election.cc:290] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.539544 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92576756ec574ceaa4c81f435cb72301" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.539841 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.540050 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:33.540110 23604 leader_election.cc:343] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.540218 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.540369 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.541038 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92576756ec574ceaa4c81f435cb72301" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.541725 25172 raft_consensus.cc:493] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.541854 25172 raft_consensus.cc:515] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:33.542351 23605 leader_election.cc:343] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.542451 23605 leader_election.cc:304] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.542851 25264 raft_consensus.cc:2749] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.543125 25172 leader_election.cc:290] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.545205 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "377de4ea0bd6489284fac6b0fab52c8c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.550542 23787 tablet_bootstrap.cc:654] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.551510 23790 tablet_bootstrap.cc:492] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:33.551702 23790 ts_tablet_manager.cc:1403] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20251023 20:15:33.552528 23790 raft_consensus.cc:359] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.552733 23790 raft_consensus.cc:385] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.552824 23790 raft_consensus.cc:740] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.553124 23790 consensus_queue.cc:260] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.554513 23790 ts_tablet_manager.cc:1434] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:33.554828 23790 tablet_bootstrap.cc:492] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:33.555452 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:33.556521 23333 consensus_peers.cc:597] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 514cb3b1ed024041ac160a7d9acff618. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:33.556792 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "377de4ea0bd6489284fac6b0fab52c8c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.556973 23669 raft_consensus.cc:2468] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.557364 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.558087 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.558524 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.559567 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.560531 23786 tablet_bootstrap.cc:492] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.560734 23786 ts_tablet_manager.cc:1403] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.001s
I20251023 20:15:33.561378 25144 raft_consensus.cc:493] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.561504 25144 raft_consensus.cc:515] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.562116 25144 leader_election.cc:290] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.561486 23786 raft_consensus.cc:359] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.562788 23786 raft_consensus.cc:385] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.562896 23786 raft_consensus.cc:740] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.563148 23786 consensus_queue.cc:260] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.563200 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94cd8880a78c4b4eb0c19ba5944b12f2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.563386 23669 raft_consensus.cc:2468] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:33.563882 23470 leader_election.cc:304] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.564214 25144 raft_consensus.cc:2804] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.564282 25144 raft_consensus.cc:493] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.564338 25144 raft_consensus.cc:3060] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.566469 24297 raft_consensus.cc:493] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.566645 24297 raft_consensus.cc:515] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.566722 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94cd8880a78c4b4eb0c19ba5944b12f2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.566908 23392 raft_consensus.cc:2468] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:33.567200 23786 ts_tablet_manager.cc:1434] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20251023 20:15:33.567178 23333 leader_election.cc:343] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.567198 24297 leader_election.cc:290] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.567574 23786 tablet_bootstrap.cc:492] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.568238 23790 tablet_bootstrap.cc:654] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.569103 23786 tablet_bootstrap.cc:654] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.570554 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.572510 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.573242 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:33.573941 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:33.575317 23216 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 193) took 3777 ms (client timeout 1000 ms). Trace:
W20251023 20:15:33.575465 23216 rpcz_store.cc:269] 1023 20:15:29.797746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.418586 (+620840us) service_pool.cc:225] Handling call
1023 20:15:33.575257 (+3156671us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":154431,"reported_tablets":198}
I20251023 20:15:33.576076 25262 consensus_queue.cc:1048] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.576943 25262 consensus_queue.cc:1048] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.577641 25144 raft_consensus.cc:515] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.578594 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94cd8880a78c4b4eb0c19ba5944b12f2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.578748 23668 raft_consensus.cc:3060] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.581183 25144 leader_election.cc:290] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:33.601253 23470 consensus_peers.cc:597] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ed0aa7e36e64843ad6b2ad4b415832a. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:33.601709 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.602470 23786 tablet_bootstrap.cc:492] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.602653 23786 ts_tablet_manager.cc:1403] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20251023 20:15:33.602809 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.602996 23669 raft_consensus.cc:2468] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.603039 23333 leader_election.cc:304] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.603647 23333 leader_election.cc:304] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.603849 23786 raft_consensus.cc:359] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.604004 23786 raft_consensus.cc:385] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.603984 24821 raft_consensus.cc:2804] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.604050 23786 raft_consensus.cc:740] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.604059 24821 raft_consensus.cc:493] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.604121 24821 raft_consensus.cc:3060] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.604216 23786 consensus_queue.cc:260] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.604682 23786 ts_tablet_manager.cc:1434] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:33.604950 23786 tablet_bootstrap.cc:492] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.606515 23786 tablet_bootstrap.cc:654] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.609063 25236 raft_consensus.cc:2804] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.609297 25236 raft_consensus.cc:493] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.609504 25236 raft_consensus.cc:3060] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.615166 23787 tablet_bootstrap.cc:492] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.615316 23787 ts_tablet_manager.cc:1403] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.120s user 0.007s sys 0.007s
I20251023 20:15:33.616137 23787 raft_consensus.cc:359] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.616318 23787 raft_consensus.cc:385] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.616379 23787 raft_consensus.cc:740] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.616693 23787 consensus_queue.cc:260] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.619725 23790 tablet_bootstrap.cc:492] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:33.620293 23790 ts_tablet_manager.cc:1403] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20251023 20:15:33.622315 23787 ts_tablet_manager.cc:1434] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:33.622720 24780 ts_tablet_manager.cc:1929] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:33.622840 24780 log.cc:1199] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a708cd011802458aa0e2991fd4b4a2d3
I20251023 20:15:33.623188 24780 ts_tablet_manager.cc:1950] T a708cd011802458aa0e2991fd4b4a2d3 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.627099 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94cd8880a78c4b4eb0c19ba5944b12f2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:33.627274 23398 raft_consensus.cc:3060] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.628057 23787 tablet_bootstrap.cc:492] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:33.632905 23333 leader_election.cc:343] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.638545 23787 tablet_bootstrap.cc:654] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.639508 23786 tablet_bootstrap.cc:492] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.639678 23786 ts_tablet_manager.cc:1403] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20251023 20:15:33.640363 23786 raft_consensus.cc:359] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.640607 23786 raft_consensus.cc:385] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.640708 23786 raft_consensus.cc:740] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.640992 23786 consensus_queue.cc:260] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.641641 23786 ts_tablet_manager.cc:1434] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:33.641963 23786 tablet_bootstrap.cc:492] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.643282 23786 tablet_bootstrap.cc:654] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.644807 23787 tablet_bootstrap.cc:492] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.645123 23787 ts_tablet_manager.cc:1403] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251023 20:15:33.645979 23787 raft_consensus.cc:359] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.646214 23787 raft_consensus.cc:385] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.646309 23787 raft_consensus.cc:740] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.646605 23787 consensus_queue.cc:260] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.647014 25172 consensus_queue.cc:1048] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.647279 23787 ts_tablet_manager.cc:1434] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:33.647656 23787 tablet_bootstrap.cc:492] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.650167 23787 tablet_bootstrap.cc:654] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.658591 25172 consensus_queue.cc:1048] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.662112 23668 raft_consensus.cc:2468] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:33.662226 25282 ts_tablet_manager.cc:1929] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:33.662329 25282 log.cc:1199] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/e194d03a6aa946fbb2c79d91b196440b
I20251023 20:15:33.662473 24821 raft_consensus.cc:515] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.662607 25282 ts_tablet_manager.cc:1950] T e194d03a6aa946fbb2c79d91b196440b P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:33.663020 24821 leader_election.cc:290] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.663453 23470 leader_election.cc:304] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.665935 25282 tablet_replica.cc:333] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:33.666134 25282 raft_consensus.cc:2243] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.666286 25282 raft_consensus.cc:2272] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.666810 25282 ts_tablet_manager.cc:1916] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.667268 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet e194d03a6aa946fbb2c79d91b196440b (table default.Table_410a6baf06a642b18af37bebddac662f [id=75026c75215e419983355275cf38f7f3]) successfully deleted
I20251023 20:15:33.666767 25236 raft_consensus.cc:515] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.668787 25236 leader_election.cc:290] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.669517 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 161) took 4897 ms. Trace:
I20251023 20:15:33.669665 24780 rpcz_store.cc:276] 1023 20:15:28.772220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.772382 (+ 162us) service_pool.cc:225] Handling call
1023 20:15:33.501983 (+4729601us) ts_tablet_manager.cc:1151] Deleting tablet a708cd011802458aa0e2991fd4b4a2d3
1023 20:15:33.501992 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.534170 (+ 32178us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.622580 (+ 88410us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.669477 (+ 46897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4729414}
I20251023 20:15:33.670212 24780 tablet_replica.cc:333] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.670603 24780 raft_consensus.cc:2243] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.671237 24780 raft_consensus.cc:2272] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.672276 24780 ts_tablet_manager.cc:1916] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.673918 23787 tablet_bootstrap.cc:492] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.674105 25144 raft_consensus.cc:2804] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.674535 25144 raft_consensus.cc:697] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:33.675024 25144 consensus_queue.cc:237] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.675444 23787 ts_tablet_manager.cc:1403] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.006s
I20251023 20:15:33.674294 23790 raft_consensus.cc:359] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.675853 23790 raft_consensus.cc:385] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.676762 23787 raft_consensus.cc:359] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.677008 23787 raft_consensus.cc:385] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.677125 23787 raft_consensus.cc:740] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.676637 23790 raft_consensus.cc:740] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.677403 23787 consensus_queue.cc:260] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.678138 23787 ts_tablet_manager.cc:1434] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:33.678663 23787 tablet_bootstrap.cc:492] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.681998 23786 tablet_bootstrap.cc:492] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.682181 23786 ts_tablet_manager.cc:1403] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.003s
I20251023 20:15:33.682830 23786 raft_consensus.cc:359] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.683086 23786 raft_consensus.cc:385] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.683171 23786 raft_consensus.cc:740] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.683411 23786 consensus_queue.cc:260] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.683823 23786 ts_tablet_manager.cc:1434] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:33.683804 25172 consensus_queue.cc:1048] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.684244 23787 tablet_bootstrap.cc:654] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.684791 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:33.685705 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.685876 23666 raft_consensus.cc:3060] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.686756 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "377de4ea0bd6489284fac6b0fab52c8c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:33.686923 23530 raft_consensus.cc:3060] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.690557 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "377de4ea0bd6489284fac6b0fab52c8c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.690747 23668 raft_consensus.cc:3060] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.695690 23787 tablet_bootstrap.cc:492] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.695883 23787 ts_tablet_manager.cc:1403] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20251023 20:15:33.696657 23787 raft_consensus.cc:359] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.696885 23787 raft_consensus.cc:385] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.696950 23787 raft_consensus.cc:740] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.697249 23787 consensus_queue.cc:260] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.698006 23787 ts_tablet_manager.cc:1434] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:33.702087 23786 tablet_bootstrap.cc:492] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.703466 23786 tablet_bootstrap.cc:654] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.705260 23786 tablet_bootstrap.cc:492] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.705590 23786 ts_tablet_manager.cc:1403] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20251023 20:15:33.704375 23790 consensus_queue.cc:260] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.706907 23786 raft_consensus.cc:359] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.707409 23786 raft_consensus.cc:385] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.707612 23786 raft_consensus.cc:740] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.708128 23786 consensus_queue.cc:260] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.709136 23786 ts_tablet_manager.cc:1434] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:33.709652 23786 tablet_bootstrap.cc:492] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.711920 23786 tablet_bootstrap.cc:654] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:33.712368 25325 log.cc:927] Time spent T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20251023 20:15:33.712769 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet a708cd011802458aa0e2991fd4b4a2d3 (table default.Table_d2b87a7247624b0380ee21ec002b9f77 [id=f64a3298a8d44a7ba1700cac9bf91080]) successfully deleted
I20251023 20:15:33.715045 23787 tablet_bootstrap.cc:492] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:33.717741 23333 leader_election.cc:343] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.719076 23787 tablet_bootstrap.cc:654] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.719591 25236 consensus_queue.cc:1048] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.726738 25172 consensus_queue.cc:1048] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.731910 23790 ts_tablet_manager.cc:1434] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.111s user 0.003s sys 0.001s
I20251023 20:15:33.736552 23790 tablet_bootstrap.cc:492] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:33.741694 23786 tablet_bootstrap.cc:492] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.741854 23786 ts_tablet_manager.cc:1403] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.000s
I20251023 20:15:33.742641 23786 raft_consensus.cc:359] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.742837 23786 raft_consensus.cc:385] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.742892 23786 raft_consensus.cc:740] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.743094 23786 consensus_queue.cc:260] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.743598 23786 ts_tablet_manager.cc:1434] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:33.743896 23786 tablet_bootstrap.cc:492] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.745918 23786 tablet_bootstrap.cc:654] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.748039 23787 tablet_bootstrap.cc:492] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.748317 23787 ts_tablet_manager.cc:1403] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.012s
I20251023 20:15:33.749589 23787 raft_consensus.cc:359] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.758124 23786 tablet_bootstrap.cc:492] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.758366 23786 ts_tablet_manager.cc:1403] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20251023 20:15:33.758435 25276 raft_consensus.cc:493] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.758566 25276 raft_consensus.cc:515] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.759219 25276 leader_election.cc:290] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.759157 23786 raft_consensus.cc:359] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.759325 23786 raft_consensus.cc:385] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.759371 23786 raft_consensus.cc:740] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.759606 23786 consensus_queue.cc:260] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.760108 23786 ts_tablet_manager.cc:1434] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:33.760434 23786 tablet_bootstrap.cc:492] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.760645 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3befdfc85bca42bb816e11d1dd832c8c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.761349 23604 leader_election.cc:343] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.761880 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3befdfc85bca42bb816e11d1dd832c8c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:33.762501 23605 leader_election.cc:343] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.762610 23605 leader_election.cc:304] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.762995 25276 raft_consensus.cc:2749] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.758037 23787 raft_consensus.cc:385] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.763446 23787 raft_consensus.cc:740] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.764428 23787 consensus_queue.cc:260] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.765270 23398 raft_consensus.cc:2468] T 94cd8880a78c4b4eb0c19ba5944b12f2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:33.765628 23666 raft_consensus.cc:2468] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:33.765980 23530 raft_consensus.cc:2468] T 377de4ea0bd6489284fac6b0fab52c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:33.766584 23787 ts_tablet_manager.cc:1434] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251023 20:15:33.767659 23787 tablet_bootstrap.cc:492] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.768301 23786 tablet_bootstrap.cc:654] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.773502 23786 tablet_bootstrap.cc:492] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.774084 23790 tablet_bootstrap.cc:654] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.775136 23786 ts_tablet_manager.cc:1403] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
W20251023 20:15:33.785969 23204 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0eb7935c4cf64e2f802ca12c1f63317b consensus state has no leader
I20251023 20:15:33.785576 23787 tablet_bootstrap.cc:654] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.784339 23786 raft_consensus.cc:359] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.786753 23786 raft_consensus.cc:385] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.787487 23786 raft_consensus.cc:740] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.790032 23786 consensus_queue.cc:260] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.794499 25172 consensus_queue.cc:1048] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:33.800294 23333 leader_election.cc:304] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.801222 25144 raft_consensus.cc:493] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.801412 25144 raft_consensus.cc:515] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.802505 25144 leader_election.cc:290] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.802620 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b766ff2dff34a8f80f70591dba54017" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.802809 23666 raft_consensus.cc:2468] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:33.803170 23470 leader_election.cc:304] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.803560 25144 raft_consensus.cc:2804] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.803664 25144 raft_consensus.cc:493] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.803750 25144 raft_consensus.cc:3060] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.804406 25172 raft_consensus.cc:2804] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.804522 25172 raft_consensus.cc:697] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:33.804776 25172 consensus_queue.cc:237] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.810364 23787 tablet_bootstrap.cc:492] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.810747 23787 ts_tablet_manager.cc:1403] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.007s
I20251023 20:15:33.811319 25262 raft_consensus.cc:493] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.811443 25262 raft_consensus.cc:515] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.812000 25262 leader_election.cc:290] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.812017 23787 raft_consensus.cc:359] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.812754 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.812960 23786 ts_tablet_manager.cc:1434] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.035s user 0.002s sys 0.002s
I20251023 20:15:33.813205 23669 raft_consensus.cc:2393] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:33.812919 23787 raft_consensus.cc:385] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.813836 23787 raft_consensus.cc:740] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.814337 23786 tablet_bootstrap.cc:492] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.814286 23787 consensus_queue.cc:260] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.815295 23787 ts_tablet_manager.cc:1434] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251023 20:15:33.816180 23787 tablet_bootstrap.cc:492] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.827127 23787 tablet_bootstrap.cc:654] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.829038 25276 raft_consensus.cc:493] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.829193 25276 raft_consensus.cc:515] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.829823 25276 leader_election.cc:290] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:33.830480 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "473324b22da14618a84aa1136d081993" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.830819 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "473324b22da14618a84aa1136d081993" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:33.830983 23604 leader_election.cc:343] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:33.831441 23605 leader_election.cc:343] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.831555 23605 leader_election.cc:304] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.829319 23786 tablet_bootstrap.cc:654] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.832051 25276 raft_consensus.cc:2749] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.839067 25172 raft_consensus.cc:493] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.839257 25172 raft_consensus.cc:515] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.839888 25172 leader_election.cc:290] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.845971 25236 raft_consensus.cc:493] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.846158 25236 raft_consensus.cc:515] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.846799 25236 leader_election.cc:290] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.849720 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b766ff2dff34a8f80f70591dba54017" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.849890 23398 raft_consensus.cc:2468] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:33.850797 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "559a70cf17ad4145acdb3033e01ad217" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.852288 23333 leader_election.cc:304] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.853322 23790 tablet_bootstrap.cc:492] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:33.855341 23790 ts_tablet_manager.cc:1403] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.120s user 0.013s sys 0.000s
I20251023 20:15:33.856894 23790 raft_consensus.cc:359] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.857272 23790 raft_consensus.cc:385] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.857419 23790 raft_consensus.cc:740] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.857858 23790 consensus_queue.cc:260] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.858973 23790 ts_tablet_manager.cc:1434] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:33.859086 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:33.859299 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:33.859510 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:33.859671 23790 tablet_bootstrap.cc:492] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:33.861482 23668 raft_consensus.cc:2468] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:33.862025 25282 ts_tablet_manager.cc:1929] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:33.862118 25282 log.cc:1199] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/7e36bb5e61d34b35a373bd6a51a7ac5e
I20251023 20:15:33.862387 25282 ts_tablet_manager.cc:1950] T 7e36bb5e61d34b35a373bd6a51a7ac5e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:33.862694 25144 raft_consensus.cc:515] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.863343 25144 leader_election.cc:290] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.864280 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b766ff2dff34a8f80f70591dba54017" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.864432 23668 raft_consensus.cc:3060] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.866906 23668 raft_consensus.cc:2468] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:33.867468 23470 leader_election.cc:304] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.867954 25144 raft_consensus.cc:493] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.868180 25144 raft_consensus.cc:515] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.868798 25236 raft_consensus.cc:2804] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.868932 25236 raft_consensus.cc:697] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:33.869161 25144 leader_election.cc:290] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.869287 25144 raft_consensus.cc:2804] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:33.869184 25236 consensus_queue.cc:237] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.869371 25144 raft_consensus.cc:697] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:33.869702 25144 consensus_queue.cc:237] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.872364 24780 ts_tablet_manager.cc:1929] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:33.872493 24780 log.cc:1199] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/283166bbd8004ce3bca9f2ecbdbcbf48
I20251023 20:15:33.872838 24780 ts_tablet_manager.cc:1950] T 283166bbd8004ce3bca9f2ecbdbcbf48 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:33.873589 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.873790 23668 raft_consensus.cc:2468] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:33.874264 23470 leader_election.cc:304] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:33.874639 25167 raft_consensus.cc:2804] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.874709 25167 raft_consensus.cc:493] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.874766 25167 raft_consensus.cc:3060] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.884594 23786 tablet_bootstrap.cc:492] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.886377 23786 ts_tablet_manager.cc:1403] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.006s
I20251023 20:15:33.860003 23210 catalog_manager.cc:5649] T 8a630b83f614447ebf2a08851d6b827d P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:33.886958 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:33.887285 23210 catalog_manager.cc:5649] T 2a1bf4f01fbc4645bcf8392f56f853b2 P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:33.891222 23786 raft_consensus.cc:359] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.892338 23786 raft_consensus.cc:385] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.892606 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.894403 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72d1dde8b0ac4812869ef233c18e550f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.894591 23669 raft_consensus.cc:2468] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.896718 23787 tablet_bootstrap.cc:492] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.896781 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.896989 23668 raft_consensus.cc:2468] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.897024 23787 ts_tablet_manager.cc:1403] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.005s
I20251023 20:15:33.887785 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:33.898902 23790 tablet_bootstrap.cc:654] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.899525 23210 catalog_manager.cc:5649] T 6e4189623c7a4ca1bdf73f71663e2b77 P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:33.900173 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:33.900481 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:33.900125 23787 raft_consensus.cc:359] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.900820 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:33.900996 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:33.900977 23787 raft_consensus.cc:385] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.901221 23210 catalog_manager.cc:5649] T 04f7dffd94f3442ea20045a512d7bd65 P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:33.901558 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:33.901420 23787 raft_consensus.cc:740] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.901788 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:33.902380 23787 consensus_queue.cc:260] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.904942 25276 raft_consensus.cc:493] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.905181 25276 raft_consensus.cc:515] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.905718 25276 leader_election.cc:290] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.906286 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.906831 23787 ts_tablet_manager.cc:1434] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
W20251023 20:15:33.910970 23605 leader_election.cc:343] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.912925 25264 raft_consensus.cc:493] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.913051 25264 raft_consensus.cc:515] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.913662 25264 leader_election.cc:290] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:33.914764 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b220ceff0924b26ae8a54dfb8705456" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.892844 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72d1dde8b0ac4812869ef233c18e550f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.915524 23531 raft_consensus.cc:2468] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:33.900509 23471 leader_election.cc:304] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:33.892783 23786 raft_consensus.cc:740] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.906883 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:33.914302 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b220ceff0924b26ae8a54dfb8705456" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:33.916848 23604 leader_election.cc:343] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.916975 23604 leader_election.cc:304] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.917203 25276 raft_consensus.cc:2749] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:33.917371 23605 leader_election.cc:343] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:33.917511 23604 leader_election.cc:343] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:33.917595 23604 leader_election.cc:304] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:33.917781 25144 raft_consensus.cc:3060] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.918926 25264 raft_consensus.cc:2749] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.917730 23786 consensus_queue.cc:260] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.924259 23787 tablet_bootstrap.cc:492] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.928859 23333 leader_election.cc:304] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.930403 25262 raft_consensus.cc:493] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.930541 25262 raft_consensus.cc:515] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.930441 23786 ts_tablet_manager.cc:1434] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.043s user 0.002s sys 0.001s
I20251023 20:15:33.931128 25262 leader_election.cc:290] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.931306 25236 raft_consensus.cc:2804] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.931447 25236 raft_consensus.cc:493] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.931653 25236 raft_consensus.cc:3060] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.931895 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.932085 25282 tablet_replica.cc:333] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:33.932116 23668 raft_consensus.cc:2393] T 21a508e875c449c5831968562263f1bf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:33.932293 25282 raft_consensus.cc:2243] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:33.932421 25282 raft_consensus.cc:2272] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.932919 25282 ts_tablet_manager.cc:1916] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.901959 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:33.924288 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 7e36bb5e61d34b35a373bd6a51a7ac5e (table default.Table_736b5fe2d49043f896fa85aeed517265 [id=9e1ee844cf9e41e29059059fb01fb5aa]) successfully deleted
W20251023 20:15:33.933693 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 9ed0aa7e36e64843ad6b2ad4b415832a because the tablet was not found. No further retry: Not found: Tablet not found: 9ed0aa7e36e64843ad6b2ad4b415832a
I20251023 20:15:33.934473 23790 tablet_bootstrap.cc:492] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:33.936226 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:33.936422 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:33.936637 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
W20251023 20:15:33.936770 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 514cb3b1ed024041ac160a7d9acff618 because the tablet was not found. No further retry: Not found: Tablet not found: 514cb3b1ed024041ac160a7d9acff618
W20251023 20:15:33.937059 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 72408899f710470baaf6fdb268cb04c8 because the tablet was not found. No further retry: Not found: Tablet not found: 72408899f710470baaf6fdb268cb04c8
W20251023 20:15:33.937294 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet c9bfbf0acdc84f2f8fe553694ee9897c because the tablet was not found. No further retry: Not found: Tablet not found: c9bfbf0acdc84f2f8fe553694ee9897c
W20251023 20:15:33.937469 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 05933cb9433d48d285b2db82b289bdc8 because the tablet was not found. No further retry: Not found: Tablet not found: 05933cb9433d48d285b2db82b289bdc8
W20251023 20:15:33.937629 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d2ae19cc80194363b8a9c96fbafb4ebe because the tablet was not found. No further retry: Not found: Tablet not found: d2ae19cc80194363b8a9c96fbafb4ebe
I20251023 20:15:33.936812 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:33.939692 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:33.939910 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:33.939994 23787 tablet_bootstrap.cc:654] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.941363 23786 tablet_bootstrap.cc:492] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:33.940986 23333 leader_election.cc:304] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:33.942399 23790 ts_tablet_manager.cc:1403] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.000s
I20251023 20:15:33.943058 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:33.943261 23210 catalog_manager.cc:5649] T 19c58a406d564b048fb16a85aa0b754b P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } }
I20251023 20:15:33.943625 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:33.944041 25172 raft_consensus.cc:2804] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:33.944118 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:33.944249 25172 raft_consensus.cc:493] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:33.944433 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:33.944418 25172 raft_consensus.cc:3060] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.945044 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:33.949649 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:33.950408 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:33.951050 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:33.951313 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b766ff2dff34a8f80f70591dba54017" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:33.951483 23398 raft_consensus.cc:3060] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.952912 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:33.953032 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.953372 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:33.953656 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:33.953905 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:33.954332 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:33.954557 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:33.954990 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:33.955029 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21a508e875c449c5831968562263f1bf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:33.955703 25262 raft_consensus.cc:493] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.955853 25262 raft_consensus.cc:515] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.956437 25262 leader_election.cc:290] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.957162 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:33.957320 24981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Bootstrap replayed 1/1 log segments. Stats: ops{read=658 overwritten=0 applied=658 ignored=0} inserts{seen=602 ignored=0} mutations{seen=762 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251023 20:15:33.957396 23668 raft_consensus.cc:2393] T efad9b4a8e2e446f917df7ca26fbf789 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:33.955363 23210 catalog_manager.cc:5649] T 165d62dd42da460faccec706f08272a1 P ea6853efbbcc484eb7529c7e9b37a47a reported cstate change: term changed from 1 to 2, leader changed from d220aba7c9c0400398ffde917376ee48 (127.22.127.129) to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 2 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:33.959985 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:33.960336 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
W20251023 20:15:33.960950 23949 leader_election.cc:336] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509): Timed out: RequestConsensusVote RPC to 127.22.127.189:33509 timed out after 0.621s (SENT)
I20251023 20:15:33.961154 23949 leader_election.cc:304] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd; no voters: 690ffde75cd745b6a87925967d54471a, ae5a310b4a514c0f812bda133c756062
I20251023 20:15:33.959782 23790 raft_consensus.cc:359] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.963587 25167 raft_consensus.cc:515] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.964221 25167 leader_election.cc:290] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.966872 25144 raft_consensus.cc:2749] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:33.967118 23534 raft_consensus.cc:2393] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:33.969326 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:33.969522 23668 raft_consensus.cc:3060] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.963102 23790 raft_consensus.cc:385] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.978729 23471 leader_election.cc:304] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:33.979184 25144 raft_consensus.cc:3060] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:33.982189 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 162) took 5209 ms. Trace:
I20251023 20:15:33.982851 24780 rpcz_store.cc:276] 1023 20:15:28.772735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.772902 (+ 167us) service_pool.cc:225] Handling call
1023 20:15:33.670143 (+4897241us) ts_tablet_manager.cc:1151] Deleting tablet 283166bbd8004ce3bca9f2ecbdbcbf48
1023 20:15:33.670152 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:33.780937 (+110785us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.872223 (+ 91286us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:33.982162 (+109939us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4897087}
I20251023 20:15:33.983217 24780 tablet_replica.cc:333] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:33.983397 24780 raft_consensus.cc:2243] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:33.983772 24780 raft_consensus.cc:2272] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:33.984364 24780 ts_tablet_manager.cc:1916] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:33.986446 23786 tablet_bootstrap.cc:654] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.985389 25355 tablet.cc:1722] T b680b1cbc103403b91a1104248ecbdcf P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:33.988739 23787 tablet_bootstrap.cc:492] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:33.988869 23787 ts_tablet_manager.cc:1403] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.002s
I20251023 20:15:33.989617 23787 raft_consensus.cc:359] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.989777 23787 raft_consensus.cc:385] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:33.989826 23787 raft_consensus.cc:740] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:33.989995 23787 consensus_queue.cc:260] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.990473 23787 ts_tablet_manager.cc:1434] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:33.990669 24981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: Bootstrap complete.
I20251023 20:15:33.994251 24297 raft_consensus.cc:493] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:33.994397 24297 raft_consensus.cc:515] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:33.995008 24297 leader_election.cc:290] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:33.997365 23787 tablet_bootstrap.cc:492] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:33.998452 23786 tablet_bootstrap.cc:492] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:33.998713 23786 ts_tablet_manager.cc:1403] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20251023 20:15:33.999051 23787 tablet_bootstrap.cc:654] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:33.999788 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:34.000284 23786 raft_consensus.cc:359] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.000978 23786 raft_consensus.cc:385] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.001137 23786 raft_consensus.cc:740] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.001255 25040 consensus_queue.cc:1048] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.001590 23786 consensus_queue.cc:260] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.003211 25357 tablet.cc:1722] T b680b1cbc103403b91a1104248ecbdcf P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:34.004609 25358 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 20:15:34.009249 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:34.010015 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "efad9b4a8e2e446f917df7ca26fbf789" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.013334 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.013522 23669 raft_consensus.cc:2468] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:34.017566 23949 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.189:33509, user_credentials={real_user=slave}} blocked reactor thread for 56627.4us
I20251023 20:15:34.011933 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.014676 25167 consensus_queue.cc:1048] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251023 20:15:34.002956 25356 tablet.cc:1722] T b680b1cbc103403b91a1104248ecbdcf P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:34.002646 23786 ts_tablet_manager.cc:1434] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20251023 20:15:34.021690 23333 leader_election.cc:343] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.022292 23786 tablet_bootstrap.cc:492] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.024094 23471 leader_election.cc:304] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.025101 25167 raft_consensus.cc:3060] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.027593 23333 leader_election.cc:304] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.029016 25040 raft_consensus.cc:2804] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.029119 25040 raft_consensus.cc:493] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.029179 25040 raft_consensus.cc:3060] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.036530 25172 raft_consensus.cc:515] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.037340 23392 raft_consensus.cc:2393] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:34.037971 23787 tablet_bootstrap.cc:492] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.038121 23787 ts_tablet_manager.cc:1403] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.007s
I20251023 20:15:34.038746 25172 leader_election.cc:290] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.038890 23787 raft_consensus.cc:359] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.039093 23787 raft_consensus.cc:385] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.010269 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 283166bbd8004ce3bca9f2ecbdbcbf48 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:34.039856 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:34.040460 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:33.982863 23790 raft_consensus.cc:740] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.041601 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:34.042486 23790 consensus_queue.cc:260] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.043037 23790 ts_tablet_manager.cc:1434] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.100s user 0.002s sys 0.000s
I20251023 20:15:34.043319 23790 tablet_bootstrap.cc:492] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:34.043253 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:34.043460 23534 raft_consensus.cc:2393] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:34.039172 23787 raft_consensus.cc:740] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.044178 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f03df77c344b4736a30725e4d9f72a35" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:34.043782 23786 tablet_bootstrap.cc:654] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.044349 23787 consensus_queue.cc:260] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.048264 23787 ts_tablet_manager.cc:1434] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251023 20:15:34.048606 23787 tablet_bootstrap.cc:492] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.051963 23787 tablet_bootstrap.cc:654] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.057461 23333 leader_election.cc:304] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.059865 25172 raft_consensus.cc:2749] T f03df77c344b4736a30725e4d9f72a35 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.061467 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:34.062114 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:34.062434 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:34.062702 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:34.062896 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:34.064301 25144 raft_consensus.cc:2749] T 21a508e875c449c5831968562263f1bf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.066792 25167 raft_consensus.cc:2749] T efad9b4a8e2e446f917df7ca26fbf789 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.069445 25236 raft_consensus.cc:515] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.070086 25236 leader_election.cc:290] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.070520 23668 raft_consensus.cc:2468] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:34.072335 23470 leader_election.cc:304] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.072783 25167 raft_consensus.cc:2804] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:34.072911 25167 raft_consensus.cc:697] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:34.073156 25167 consensus_queue.cc:237] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.079627 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72d1dde8b0ac4812869ef233c18e550f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:34.079891 23668 raft_consensus.cc:3060] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.082111 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72d1dde8b0ac4812869ef233c18e550f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:34.082276 23531 raft_consensus.cc:3060] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.085534 25172 consensus_queue.cc:1048] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.088690 25172 raft_consensus.cc:493] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea6853efbbcc484eb7529c7e9b37a47a)
I20251023 20:15:34.088990 25172 raft_consensus.cc:515] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.090408 25172 leader_election.cc:290] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.092602 23790 tablet_bootstrap.cc:654] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.063207 23210 catalog_manager.cc:5741] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:34.106679 25359 raft_consensus.cc:493] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.106815 25359 raft_consensus.cc:515] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:34.101209 23333 leader_election.cc:343] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:34.107381 25359 leader_election.cc:290] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:34.108073 23333 leader_election.cc:343] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:34.108176 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40c659c273c449939d0dd708d39b393f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.108744 23604 leader_election.cc:343] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.109010 23333 leader_election.cc:304] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.109372 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40c659c273c449939d0dd708d39b393f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.109534 25235 raft_consensus.cc:2749] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:34.109871 23605 leader_election.cc:343] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.109972 23605 leader_election.cc:304] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.111617 25359 raft_consensus.cc:2749] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:34.134028 25365 log.cc:927] Time spent T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20251023 20:15:34.136425 25355 tablet.cc:1722] T a04c65155ce048cb82b150cbe20be849 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.140837 25359 raft_consensus.cc:493] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.140995 25359 raft_consensus.cc:515] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.141703 25359 leader_election.cc:290] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:34.142202 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ae1a94b011412c851cd96946744fc4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.143451 23605 leader_election.cc:343] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.143854 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ae1a94b011412c851cd96946744fc4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.144362 23604 leader_election.cc:343] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.144465 23604 leader_election.cc:304] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.144896 25359 raft_consensus.cc:2749] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.149542 25384 tablet.cc:1722] T a04c65155ce048cb82b150cbe20be849 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.153867 25262 consensus_queue.cc:1048] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.162640 23531 raft_consensus.cc:2468] T 72d1dde8b0ac4812869ef233c18e550f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:34.162761 25282 ts_tablet_manager.cc:1929] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:34.162882 25282 log.cc:1199] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/01fa5f9c700b458ea147373d78551573
I20251023 20:15:34.164269 25282 ts_tablet_manager.cc:1950] T 01fa5f9c700b458ea147373d78551573 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:34.164500 23668 raft_consensus.cc:2468] T 72d1dde8b0ac4812869ef233c18e550f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:34.165498 25282 tablet_replica.cc:333] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:34.165683 25282 raft_consensus.cc:2243] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:34.165730 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 01fa5f9c700b458ea147373d78551573 (table default.Table_6d6d11ad70eb49409020a15552192008 [id=9ecd31de55764cc9ab7f83b1a614302f]) successfully deleted
I20251023 20:15:34.165819 25282 raft_consensus.cc:2272] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.166349 25282 ts_tablet_manager.cc:1916] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:34.160998 25387 tablet.cc:1722] T 0ca01bea2bb146ca8797a9592a840a81 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.166105 25040 raft_consensus.cc:515] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.184154 25040 leader_election.cc:290] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.165136 24780 ts_tablet_manager.cc:1929] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:34.184603 24780 log.cc:1199] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d951134f6773410386bc57f7f90310c3
I20251023 20:15:34.187180 25389 tablet.cc:1722] T 2fea0afc8f63497d8b5cdf2352a20267 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:34.188714 24780 ts_tablet_manager.cc:1950] T d951134f6773410386bc57f7f90310c3 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:34.161403 25379 tablet.cc:1722] T 2fea0afc8f63497d8b5cdf2352a20267 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:34.191385 25356 tablet.cc:1722] T 0ca01bea2bb146ca8797a9592a840a81 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.180012 23333 leader_election.cc:304] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:34.194891 25040 raft_consensus.cc:2804] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:34.195003 25040 raft_consensus.cc:697] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:34.195232 25040 consensus_queue.cc:237] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.195778 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:34.195982 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:34.196245 23216 catalog_manager.cc:5649] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.203164 25172 consensus_queue.cc:1048] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.183168 23790 tablet_bootstrap.cc:492] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:34.208307 25388 tablet.cc:1722] T 0ca01bea2bb146ca8797a9592a840a81 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.209574 23786 tablet_bootstrap.cc:492] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.209735 23786 ts_tablet_manager.cc:1403] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.188s user 0.006s sys 0.000s
I20251023 20:15:34.207238 23790 ts_tablet_manager.cc:1403] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.164s user 0.005s sys 0.001s
I20251023 20:15:34.210745 23786 raft_consensus.cc:359] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.210938 23786 raft_consensus.cc:385] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.211084 23786 raft_consensus.cc:740] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.211350 23786 consensus_queue.cc:260] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.211735 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:34.211866 23786 ts_tablet_manager.cc:1434] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.211915 23665 raft_consensus.cc:3060] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.214841 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:34.218176 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:34.218966 23216 catalog_manager.cc:5649] T c3d238e071cb4d49b173febd90bc909b P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.219874 23790 raft_consensus.cc:359] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.222028 23790 raft_consensus.cc:385] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.222929 23790 raft_consensus.cc:740] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.225467 23216 catalog_manager.cc:5649] T 21a508e875c449c5831968562263f1bf P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.226671 23216 catalog_manager.cc:5649] T e5af107f9a144edb9080b7135ab8aeb9 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc6522db11e4960a9aad84435371a32 (127.22.127.131). New cstate: current_term: 1 leader_uuid: "9bc6522db11e4960a9aad84435371a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } }
I20251023 20:15:34.227490 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:34.227973 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:34.225749 25394 tablet.cc:1722] T a04c65155ce048cb82b150cbe20be849 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.226140 23790 consensus_queue.cc:260] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.229405 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:34.229923 23216 catalog_manager.cc:5649] T 1edaed8c82da4e36a312477b15017460 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.231307 23787 tablet_bootstrap.cc:492] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.231480 23787 ts_tablet_manager.cc:1403] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.183s user 0.008s sys 0.028s
I20251023 20:15:34.232015 23786 tablet_bootstrap.cc:492] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.232324 23787 raft_consensus.cc:359] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.232600 23787 raft_consensus.cc:385] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.232678 23787 raft_consensus.cc:740] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.232906 23787 consensus_queue.cc:260] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.233521 23787 ts_tablet_manager.cc:1434] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.233820 23787 tablet_bootstrap.cc:492] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.222234 24981 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:34.236900 24981 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5a310b4a514c0f812bda133c756062, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.240427 23216 catalog_manager.cc:5649] T ac4d82d024fc491ab0dbe04147cb156c P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.241263 23216 catalog_manager.cc:5649] T eb5a6d9a4509431f97b69a3b138b5829 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.244755 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:34.245067 23216 catalog_manager.cc:5649] T 124703edbf08408a9ca69732674c0ad3 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:34.245471 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:34.248253 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:34.248492 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:34.249039 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:34.249265 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:34.249467 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:34.249668 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:34.249883 23216 catalog_manager.cc:5649] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.250233 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:34.250524 23216 catalog_manager.cc:5649] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.251017 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:34.251287 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:34.251627 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
W20251023 20:15:34.252892 23333 leader_election.cc:343] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.253419 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:34.253999 25357 tablet.cc:1722] T 0150ab3778c34c77a7e772dd6e82b32f P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:34.255048 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:34.255476 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:34.255970 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:34.256291 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:34.258302 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:34.258558 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:34.258664 25282 ts_tablet_manager.cc:1929] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:34.258772 25282 log.cc:1199] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/40c659c273c449939d0dd708d39b393f
I20251023 20:15:34.259033 25282 ts_tablet_manager.cc:1950] T 40c659c273c449939d0dd708d39b393f P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:34.260365 23398 raft_consensus.cc:2468] T 9b766ff2dff34a8f80f70591dba54017 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:34.260525 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 163) took 5487 ms. Trace:
I20251023 20:15:34.260628 24780 rpcz_store.cc:276] 1023 20:15:28.773285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.773441 (+ 156us) service_pool.cc:225] Handling call
1023 20:15:33.983170 (+5209729us) ts_tablet_manager.cc:1151] Deleting tablet d951134f6773410386bc57f7f90310c3
1023 20:15:33.983180 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:34.074582 (+ 91402us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:34.165004 (+ 90422us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:34.260509 (+ 95505us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5209555}
I20251023 20:15:34.260744 23665 raft_consensus.cc:2468] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:34.260960 24780 tablet_replica.cc:333] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:34.261059 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:34.261155 24780 raft_consensus.cc:2243] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:34.261511 24780 raft_consensus.cc:2272] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.261708 25359 raft_consensus.cc:493] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.261833 25359 raft_consensus.cc:515] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.262180 24780 ts_tablet_manager.cc:1916] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:34.262434 25359 leader_election.cc:290] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.263389 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39a62c97e0a3459e91c24c1c2f92389a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.263936 23605 leader_election.cc:343] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.264207 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39a62c97e0a3459e91c24c1c2f92389a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.264688 23604 leader_election.cc:343] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.264787 23604 leader_election.cc:304] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.265197 25359 raft_consensus.cc:2749] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.266244 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:34.243745 24981 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 658, Last appended: 1.658, Last appended by leader: 658, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:34.268936 23786 tablet_bootstrap.cc:654] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.270903 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:34.271245 23216 catalog_manager.cc:5649] T 385900c60d584c15b702f186ecfbc036 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.272172 23790 ts_tablet_manager.cc:1434] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.061s user 0.001s sys 0.002s
I20251023 20:15:34.273769 23216 catalog_manager.cc:5649] T 975fc4848c8f4db4bfd2dad9ac620d83 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:34.274761 23787 tablet_bootstrap.cc:654] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.275153 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:34.277606 25384 tablet.cc:1722] T 744d31fccbbe4fa59c56e24136e3b2b9 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.276566 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d951134f6773410386bc57f7f90310c3 (table default.Table_deb8b605c159407ab25c527a038c8d44 [id=7313119e56d94c05b9d8a31709af6ff6]) successfully deleted
I20251023 20:15:34.278976 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:34.279302 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:34.279508 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:34.279626 25356 tablet.cc:1722] T 0150ab3778c34c77a7e772dd6e82b32f P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:34.279881 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:34.280046 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:34.280537 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:34.280730 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:34.281262 23216 catalog_manager.cc:5649] T a570f339890946b7ab20c15de68462d5 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:34.280994 23790 tablet_bootstrap.cc:492] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:34.287457 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:34.289192 23216 catalog_manager.cc:5649] T 5407a850ec7045158f53cb8abf6fcb58 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:34.291280 23790 tablet_bootstrap.cc:654] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.292999 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:34.294340 23786 tablet_bootstrap.cc:492] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.295545 23786 ts_tablet_manager.cc:1403] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.002s
I20251023 20:15:34.299512 24981 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: configured and running, proceeding with master startup.
I20251023 20:15:34.301187 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:34.303257 23333 leader_election.cc:304] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.299585 24924 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 755 } } ignore_live_leader: false dest_uuid: "ae5a310b4a514c0f812bda133c756062" is_pre_election: true
I20251023 20:15:34.299942 24922 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 755 } } ignore_live_leader: false dest_uuid: "ae5a310b4a514c0f812bda133c756062" is_pre_election: true
I20251023 20:15:34.303828 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:34.302289 24925 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 1 FOLLOWER]: Advancing to term 2
W20251023 20:15:34.304109 24922 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52066 (request call id 1) took 965 ms (client timeout 621 ms). Trace:
W20251023 20:15:34.304308 24922 rpcz_store.cc:269] 1023 20:15:33.338955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.339036 (+ 81us) service_pool.cc:225] Handling call
1023 20:15:34.304047 (+965011us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 20:15:34.305217 24924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52066 (request call id 0) took 1551 ms (client timeout 621 ms). Trace:
I20251023 20:15:34.305305 25235 raft_consensus.cc:2804] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
W20251023 20:15:34.305388 24924 rpcz_store.cc:269] 1023 20:15:32.753535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:32.753607 (+ 72us) service_pool.cc:225] Handling call
1023 20:15:34.305171 (+1551564us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:34.305485 25235 raft_consensus.cc:697] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:34.305521 25359 raft_consensus.cc:493] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.305665 25359 raft_consensus.cc:515] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.305820 25235 consensus_queue.cc:237] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.306252 25359 leader_election.cc:290] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.306795 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d71f0cfdb6449fbd74b854c15326d6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.306949 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d71f0cfdb6449fbd74b854c15326d6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.307287 23605 leader_election.cc:343] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.307457 23216 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
W20251023 20:15:34.307487 23604 leader_election.cc:343] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.307581 23604 leader_election.cc:304] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.307934 25359 raft_consensus.cc:2749] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.296578 25399 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:34.309118 25399 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:34.309993 23786 raft_consensus.cc:359] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.310148 23786 raft_consensus.cc:385] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.310189 23786 raft_consensus.cc:740] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.310355 23786 consensus_queue.cc:260] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.310781 23786 ts_tablet_manager.cc:1434] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251023 20:15:34.317047 25172 consensus_queue.cc:1048] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.319530 25355 tablet.cc:1722] T 744d31fccbbe4fa59c56e24136e3b2b9 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.320315 23786 tablet_bootstrap.cc:492] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.320708 25401 tablet.cc:1722] T 2fea0afc8f63497d8b5cdf2352a20267 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:34.324393 25172 consensus_queue.cc:1048] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.324993 23786 tablet_bootstrap.cc:654] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.326498 23534 raft_consensus.cc:3060] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.334950 25359 raft_consensus.cc:493] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.335129 25359 raft_consensus.cc:515] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.335745 25359 leader_election.cc:290] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.336733 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a6a2d2b25824dbdafaf2a9cabb0f061" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.336880 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a6a2d2b25824dbdafaf2a9cabb0f061" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.337327 23605 leader_election.cc:343] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:34.337602 23604 leader_election.cc:343] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.337698 23604 leader_election.cc:304] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.338404 25276 raft_consensus.cc:2749] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.338344 25406 tablet.cc:1722] T 0150ab3778c34c77a7e772dd6e82b32f P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251023 20:15:34.339916 25276 raft_consensus.cc:493] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.340031 25276 raft_consensus.cc:515] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.340597 25276 leader_election.cc:290] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.341235 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e2fec8f4e14036af714427481f2f36" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.341794 23605 leader_election.cc:343] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.341813 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e2fec8f4e14036af714427481f2f36" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.342253 23604 leader_election.cc:343] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.342361 23604 leader_election.cc:304] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.343581 23787 tablet_bootstrap.cc:492] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.343757 23787 ts_tablet_manager.cc:1403] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.009s
I20251023 20:15:34.344599 23787 raft_consensus.cc:359] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.344796 23787 raft_consensus.cc:385] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.344887 23787 raft_consensus.cc:740] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.345228 23787 consensus_queue.cc:260] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.345515 25276 raft_consensus.cc:2749] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.346133 23786 tablet_bootstrap.cc:492] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.346195 23787 ts_tablet_manager.cc:1434] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.346469 23787 tablet_bootstrap.cc:492] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.346807 23786 ts_tablet_manager.cc:1403] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.003s
I20251023 20:15:34.348079 23787 tablet_bootstrap.cc:654] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.348905 23786 raft_consensus.cc:359] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.352061 23786 raft_consensus.cc:385] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.352408 23786 raft_consensus.cc:740] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.353149 23786 consensus_queue.cc:260] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.353924 23786 ts_tablet_manager.cc:1434] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251023 20:15:34.354672 23786 tablet_bootstrap.cc:492] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.360924 25356 tablet.cc:1722] T 744d31fccbbe4fa59c56e24136e3b2b9 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:34.363567 23787 tablet_bootstrap.cc:492] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.363837 23787 ts_tablet_manager.cc:1403] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20251023 20:15:34.364598 23787 raft_consensus.cc:359] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.365187 23787 raft_consensus.cc:385] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.365370 23787 raft_consensus.cc:740] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.362112 24830 raft_consensus.cc:493] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.362578 23786 tablet_bootstrap.cc:654] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.365716 24830 raft_consensus.cc:515] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.366508 24830 leader_election.cc:290] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.366622 24830 raft_consensus.cc:493] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.366775 24830 raft_consensus.cc:515] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.366864 23787 consensus_queue.cc:260] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.367538 25172 consensus_queue.cc:1048] T efad9b4a8e2e446f917df7ca26fbf789 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.367949 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.369874 23787 ts_tablet_manager.cc:1434] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:34.370224 23787 tablet_bootstrap.cc:492] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.371027 24830 leader_election.cc:290] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.371583 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882f7bf08e2f4b5382702c3d6eaf201e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.371759 23665 raft_consensus.cc:2468] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.372524 23470 leader_election.cc:304] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:34.372907 24830 raft_consensus.cc:2804] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.372983 24830 raft_consensus.cc:493] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.373039 24830 raft_consensus.cc:3060] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.378082 23787 tablet_bootstrap.cc:654] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.386701 23786 tablet_bootstrap.cc:492] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.387348 23786 ts_tablet_manager.cc:1403] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.006s
I20251023 20:15:34.388275 23786 raft_consensus.cc:359] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.388597 23786 raft_consensus.cc:385] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.388731 23786 raft_consensus.cc:740] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.389106 23786 consensus_queue.cc:260] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.389223 23790 tablet_bootstrap.cc:492] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:34.391991 23786 ts_tablet_manager.cc:1434] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:34.393973 23786 tablet_bootstrap.cc:492] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.394001 23787 tablet_bootstrap.cc:492] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.394119 23787 ts_tablet_manager.cc:1403] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20251023 20:15:34.394886 23787 raft_consensus.cc:359] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.395083 23787 raft_consensus.cc:385] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.395161 23787 raft_consensus.cc:740] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.395378 23787 consensus_queue.cc:260] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.395864 25167 raft_consensus.cc:493] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.395907 23787 ts_tablet_manager.cc:1434] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.396009 25167 raft_consensus.cc:515] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.396173 23787 tablet_bootstrap.cc:492] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.396549 23790 ts_tablet_manager.cc:1403] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.117s user 0.005s sys 0.000s
I20251023 20:15:34.396607 25167 leader_election.cc:290] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.397429 23790 raft_consensus.cc:359] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.397640 23790 raft_consensus.cc:385] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.397701 23790 raft_consensus.cc:740] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
W20251023 20:15:34.397677 23470 leader_election.cc:343] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 7e36bb5e61d34b35a373bd6a51a7ac5e
I20251023 20:15:34.398028 23787 tablet_bootstrap.cc:654] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.397959 23790 consensus_queue.cc:260] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.398578 23790 ts_tablet_manager.cc:1434] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:34.400015 23786 tablet_bootstrap.cc:654] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.403568 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2d4ba34cc444baf9fe725bc81478517" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.404152 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882f7bf08e2f4b5382702c3d6eaf201e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.404309 23398 raft_consensus.cc:2468] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.404690 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e36bb5e61d34b35a373bd6a51a7ac5e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.405970 24297 consensus_queue.cc:1048] T a2d4ba34cc444baf9fe725bc81478517 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.407126 23471 leader_election.cc:304] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.407454 25167 raft_consensus.cc:2749] T a2d4ba34cc444baf9fe725bc81478517 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:34.407626 23471 leader_election.cc:343] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.407716 23471 leader_election.cc:304] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.408217 25262 raft_consensus.cc:2749] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.414211 23787 tablet_bootstrap.cc:492] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.414424 23787 ts_tablet_manager.cc:1403] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20251023 20:15:34.415210 23787 raft_consensus.cc:359] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.415453 23787 raft_consensus.cc:385] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.415544 23787 raft_consensus.cc:740] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.415817 23787 consensus_queue.cc:260] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.416550 23787 ts_tablet_manager.cc:1434] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:34.416877 23787 tablet_bootstrap.cc:492] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.418553 23787 tablet_bootstrap.cc:654] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.421411 23786 tablet_bootstrap.cc:492] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.421639 23786 ts_tablet_manager.cc:1403] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20251023 20:15:34.423338 23786 raft_consensus.cc:359] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.423602 23786 raft_consensus.cc:385] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.423692 23786 raft_consensus.cc:740] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.423972 23786 consensus_queue.cc:260] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.424614 23786 ts_tablet_manager.cc:1434] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:34.425298 23786 tablet_bootstrap.cc:492] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.430318 23786 tablet_bootstrap.cc:654] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.432137 23787 tablet_bootstrap.cc:492] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.432334 23787 ts_tablet_manager.cc:1403] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251023 20:15:34.433221 23787 raft_consensus.cc:359] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.433418 23787 raft_consensus.cc:385] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.433503 23787 raft_consensus.cc:740] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.433764 23787 consensus_queue.cc:260] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.434306 23787 ts_tablet_manager.cc:1434] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.437400 23787 tablet_bootstrap.cc:492] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.438323 25276 raft_consensus.cc:493] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.438473 25276 raft_consensus.cc:515] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.439093 25276 leader_election.cc:290] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.439424 23787 tablet_bootstrap.cc:654] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.440421 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa0a46ffa626447eab4cdb956e798005" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.441012 23604 leader_election.cc:343] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.441362 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa0a46ffa626447eab4cdb956e798005" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.442018 23605 leader_election.cc:343] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.442121 23605 leader_election.cc:304] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.442515 25276 raft_consensus.cc:2749] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.442943 23790 tablet_bootstrap.cc:492] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:34.450253 25172 consensus_queue.cc:1048] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.454392 23786 tablet_bootstrap.cc:492] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.454649 23786 ts_tablet_manager.cc:1403] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.017s
I20251023 20:15:34.455565 23786 raft_consensus.cc:359] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.455844 23786 raft_consensus.cc:385] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.455945 23786 raft_consensus.cc:740] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.456131 23787 tablet_bootstrap.cc:492] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.456274 23786 consensus_queue.cc:260] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.456271 23787 ts_tablet_manager.cc:1403] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
I20251023 20:15:34.456931 23786 ts_tablet_manager.cc:1434] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:34.457190 23790 tablet_bootstrap.cc:654] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.457324 23786 tablet_bootstrap.cc:492] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.457402 23787 raft_consensus.cc:359] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.457587 23787 raft_consensus.cc:385] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.457645 23787 raft_consensus.cc:740] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.457876 23787 consensus_queue.cc:260] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.458441 23787 ts_tablet_manager.cc:1434] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.458719 23787 tablet_bootstrap.cc:492] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.459326 23786 tablet_bootstrap.cc:654] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.460737 23787 tablet_bootstrap.cc:654] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.461709 25276 raft_consensus.cc:493] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.461818 25276 raft_consensus.cc:515] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.462317 25276 leader_election.cc:290] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.463240 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1efa4fcaeb049b484ca59b12025ce9a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.463670 24830 raft_consensus.cc:515] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:34.463826 23605 leader_election.cc:343] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.464092 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1efa4fcaeb049b484ca59b12025ce9a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.464344 24830 leader_election.cc:290] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.465183 25282 tablet_replica.cc:333] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:34.465381 25282 raft_consensus.cc:2243] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:34.465533 25282 raft_consensus.cc:2272] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.467267 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 40c659c273c449939d0dd708d39b393f (table default.Table_f21169f4f8814e739bc80b279a060a39 [id=6b4bb32ab663438da2c6b86b88796979]) successfully deleted
W20251023 20:15:34.467589 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
I20251023 20:15:34.468271 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882f7bf08e2f4b5382702c3d6eaf201e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
W20251023 20:15:34.468408 23604 leader_election.cc:343] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.468513 23604 leader_election.cc:304] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.468426 23664 raft_consensus.cc:3060] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.469645 25282 ts_tablet_manager.cc:1916] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:34.477165 23786 tablet_bootstrap.cc:492] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.477401 23786 ts_tablet_manager.cc:1403] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.004s
I20251023 20:15:34.478305 23786 raft_consensus.cc:359] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.478526 23786 raft_consensus.cc:385] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.478610 23786 raft_consensus.cc:740] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.478885 23786 consensus_queue.cc:260] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.479405 23786 ts_tablet_manager.cc:1434] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.479705 23786 tablet_bootstrap.cc:492] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.481302 23786 tablet_bootstrap.cc:654] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.482070 23787 tablet_bootstrap.cc:492] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.482189 23787 ts_tablet_manager.cc:1403] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.002s
I20251023 20:15:34.482941 23787 raft_consensus.cc:359] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.483103 23787 raft_consensus.cc:385] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.483146 23787 raft_consensus.cc:740] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.483318 23787 consensus_queue.cc:260] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.483773 23787 ts_tablet_manager.cc:1434] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:34.489949 25276 raft_consensus.cc:2749] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.491240 23786 tablet_bootstrap.cc:492] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.492206 23786 ts_tablet_manager.cc:1403] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20251023 20:15:34.493520 23786 raft_consensus.cc:359] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.493836 23786 raft_consensus.cc:385] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.493917 23786 raft_consensus.cc:740] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.494167 23786 consensus_queue.cc:260] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.494586 25235 consensus_queue.cc:1048] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.494704 23786 ts_tablet_manager.cc:1434] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:34.495008 23786 tablet_bootstrap.cc:492] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.496613 23786 tablet_bootstrap.cc:654] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.499289 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:34.504401 25262 raft_consensus.cc:493] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.504545 25262 raft_consensus.cc:515] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.504683 23786 tablet_bootstrap.cc:492] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.504796 23786 ts_tablet_manager.cc:1403] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20251023 20:15:34.505232 25262 leader_election.cc:290] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.506011 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.506233 23665 raft_consensus.cc:2393] T 67e0701159c548c49e9e245820c45028 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:34.506614 23786 raft_consensus.cc:359] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.506839 23786 raft_consensus.cc:385] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.506918 23786 raft_consensus.cc:740] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.507220 23786 consensus_queue.cc:260] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.507745 23786 ts_tablet_manager.cc:1434] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:34.508045 23786 tablet_bootstrap.cc:492] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.509621 23786 tablet_bootstrap.cc:654] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.517704 23786 tablet_bootstrap.cc:492] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.517936 23786 ts_tablet_manager.cc:1403] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20251023 20:15:34.517979 23787 tablet_bootstrap.cc:492] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.519356 23786 raft_consensus.cc:359] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.519634 23786 raft_consensus.cc:385] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.519737 23786 raft_consensus.cc:740] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.519780 23787 tablet_bootstrap.cc:654] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.520038 23786 consensus_queue.cc:260] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.524641 23786 ts_tablet_manager.cc:1434] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:34.524797 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:34.527848 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:34.530385 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:34.530751 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:34.531188 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:34.531844 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:34.532353 23786 tablet_bootstrap.cc:492] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.534346 23786 tablet_bootstrap.cc:654] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.534870 25172 raft_consensus.cc:493] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.535013 25172 raft_consensus.cc:515] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.535260 25235 raft_consensus.cc:493] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.535362 25235 raft_consensus.cc:515] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.535579 25172 leader_election.cc:290] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.535821 25235 leader_election.cc:290] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.539971 23787 tablet_bootstrap.cc:492] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.540047 25359 raft_consensus.cc:493] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.540134 23787 ts_tablet_manager.cc:1403] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20251023 20:15:34.540180 25359 raft_consensus.cc:515] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.540729 25359 leader_election.cc:290] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.540957 23787 raft_consensus.cc:359] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.541136 23787 raft_consensus.cc:385] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.541182 23787 raft_consensus.cc:740] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.541239 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4f012469fa4e8e827e485287ad76dd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.541365 23787 consensus_queue.cc:260] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:34.541705 23604 leader_election.cc:343] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.541872 23787 ts_tablet_manager.cc:1434] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.542138 23787 tablet_bootstrap.cc:492] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.542310 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4f012469fa4e8e827e485287ad76dd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.542788 23605 leader_election.cc:343] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.542882 23605 leader_election.cc:304] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.543203 25276 raft_consensus.cc:2749] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.543614 23787 tablet_bootstrap.cc:654] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.545600 23536 raft_consensus.cc:3060] T 92c50c4845524163834861438291a3ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.546499 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d62dbf044acd4459b26d2e7e50f6206e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.546687 23663 raft_consensus.cc:2468] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:34.546921 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6f3abcaa81c4646bc1432bcbd1d4fcf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.547073 23670 raft_consensus.cc:2468] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:34.548712 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d62dbf044acd4459b26d2e7e50f6206e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.548892 23533 raft_consensus.cc:2468] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:34.549487 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6f3abcaa81c4646bc1432bcbd1d4fcf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.549644 23528 raft_consensus.cc:2468] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:34.549970 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "882f7bf08e2f4b5382702c3d6eaf201e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:34.550133 23398 raft_consensus.cc:3060] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.550837 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67e0701159c548c49e9e245820c45028" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.551787 23787 tablet_bootstrap.cc:492] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.551949 23787 ts_tablet_manager.cc:1403] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20251023 20:15:34.552652 23787 raft_consensus.cc:359] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.552852 23787 raft_consensus.cc:385] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.552928 23787 raft_consensus.cc:740] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.553169 23787 consensus_queue.cc:260] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.553845 23787 ts_tablet_manager.cc:1434] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.554172 23787 tablet_bootstrap.cc:492] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.555728 23787 tablet_bootstrap.cc:654] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.562399 23664 raft_consensus.cc:2468] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:34.562904 23470 leader_election.cc:304] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:34.562932 23398 raft_consensus.cc:2468] T 882f7bf08e2f4b5382702c3d6eaf201e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:34.563232 25262 raft_consensus.cc:2804] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:34.563354 25262 raft_consensus.cc:697] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:34.563608 25262 consensus_queue.cc:237] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.574131 23333 leader_election.cc:304] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:34.577296 25276 raft_consensus.cc:493] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.577450 25276 raft_consensus.cc:515] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.578055 25276 leader_election.cc:290] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.579461 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.580058 23604 leader_election.cc:343] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.580391 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.581334 23605 leader_election.cc:343] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.581436 23605 leader_election.cc:304] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.581840 25276 raft_consensus.cc:2749] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.583019 23790 tablet_bootstrap.cc:492] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:34.583213 23790 ts_tablet_manager.cc:1403] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.140s user 0.003s sys 0.002s
I20251023 20:15:34.584647 25235 raft_consensus.cc:2804] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.584734 25235 raft_consensus.cc:493] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.584791 25235 raft_consensus.cc:3060] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.588114 23790 raft_consensus.cc:359] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.588312 23790 raft_consensus.cc:385] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.588405 23790 raft_consensus.cc:740] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.588742 23790 consensus_queue.cc:260] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.589095 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:34.589339 23790 ts_tablet_manager.cc:1434] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:34.589692 23790 tablet_bootstrap.cc:492] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:34.591445 23790 tablet_bootstrap.cc:654] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.598387 23333 leader_election.cc:304] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:34.589730 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:34.599376 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:34.599681 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:34.600353 23786 tablet_bootstrap.cc:492] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.600507 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:34.600570 23786 ts_tablet_manager.cc:1403] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.068s user 0.021s sys 0.030s
I20251023 20:15:34.600751 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:34.601118 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:34.601472 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:34.601667 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:34.601979 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:34.601894 23786 raft_consensus.cc:359] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.602139 23786 raft_consensus.cc:385] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.602253 23786 raft_consensus.cc:740] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.602303 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:34.602574 23786 consensus_queue.cc:260] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.603266 23786 ts_tablet_manager.cc:1434] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:34.603965 23786 tablet_bootstrap.cc:492] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.605842 23786 tablet_bootstrap.cc:654] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.609539 24297 raft_consensus.cc:2804] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.609637 24297 raft_consensus.cc:493] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.609691 24297 raft_consensus.cc:3060] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.617930 23786 tablet_bootstrap.cc:492] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.618009 25262 raft_consensus.cc:493] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.618192 23786 ts_tablet_manager.cc:1403] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20251023 20:15:34.618155 25262 raft_consensus.cc:515] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.618849 25262 leader_election.cc:290] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.619036 23786 raft_consensus.cc:359] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.619201 23786 raft_consensus.cc:385] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.619247 23786 raft_consensus.cc:740] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.619467 23786 consensus_queue.cc:260] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.620189 23786 ts_tablet_manager.cc:1434] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:34.620383 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.620489 23786 tablet_bootstrap.cc:492] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.620656 23668 raft_consensus.cc:2468] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.621163 23470 leader_election.cc:304] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:34.621583 25262 raft_consensus.cc:2804] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.621656 25262 raft_consensus.cc:493] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.621711 25262 raft_consensus.cc:3060] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.622833 23786 tablet_bootstrap.cc:654] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.624941 25040 consensus_queue.cc:1048] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.626681 23471 leader_election.cc:304] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.627167 24830 raft_consensus.cc:3060] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.629324 25276 raft_consensus.cc:493] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.629451 25276 raft_consensus.cc:515] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.630007 25276 leader_election.cc:290] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.632117 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.632684 23787 tablet_bootstrap.cc:492] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.632668 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.632862 23787 ts_tablet_manager.cc:1403] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.079s user 0.025s sys 0.023s
I20251023 20:15:34.633695 23787 raft_consensus.cc:359] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.633852 23787 raft_consensus.cc:385] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.633898 23787 raft_consensus.cc:740] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.634114 23787 consensus_queue.cc:260] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:34.635167 23605 leader_election.cc:343] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.635620 25040 raft_consensus.cc:493] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.635751 25040 raft_consensus.cc:515] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:34.637605 23604 leader_election.cc:343] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.637708 23604 leader_election.cc:304] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.638022 25276 raft_consensus.cc:2749] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.640260 23787 ts_tablet_manager.cc:1434] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20251023 20:15:34.640661 23787 tablet_bootstrap.cc:492] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.641893 25040 leader_election.cc:290] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.642174 23787 tablet_bootstrap.cc:654] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.642274 25167 raft_consensus.cc:493] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.642385 25167 raft_consensus.cc:515] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.642944 25167 leader_election.cc:290] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.643707 23786 tablet_bootstrap.cc:492] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.643880 23786 ts_tablet_manager.cc:1403] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.015s
I20251023 20:15:34.644589 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.644631 23786 raft_consensus.cc:359] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.644764 23786 raft_consensus.cc:385] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.644827 23786 raft_consensus.cc:740] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.644847 23665 raft_consensus.cc:2393] T e18c1df87ee640148f5e827ee168fea6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:34.645015 23786 consensus_queue.cc:260] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.645649 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.646109 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18c1df87ee640148f5e827ee168fea6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.646479 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e36bb5e61d34b35a373bd6a51a7ac5e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.646644 23534 raft_consensus.cc:2468] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:34.646988 23786 ts_tablet_manager.cc:1434] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:34.647354 23786 tablet_bootstrap.cc:492] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.648769 25040 raft_consensus.cc:493] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.648927 25040 raft_consensus.cc:515] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.649535 25040 leader_election.cc:290] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.653005 24821 consensus_queue.cc:1048] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.653470 24821 consensus_queue.cc:1048] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:34.653609 23333 leader_election.cc:343] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 7e36bb5e61d34b35a373bd6a51a7ac5e
I20251023 20:15:34.653730 23787 tablet_bootstrap.cc:492] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.653882 23787 ts_tablet_manager.cc:1403] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20251023 20:15:34.654579 23787 raft_consensus.cc:359] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.654745 23787 raft_consensus.cc:385] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.654847 23787 raft_consensus.cc:740] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.655074 23787 consensus_queue.cc:260] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.655200 23333 leader_election.cc:304] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:34.655458 24821 raft_consensus.cc:2804] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.655535 24821 raft_consensus.cc:493] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.655599 24821 raft_consensus.cc:3060] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:34.655941 23471 leader_election.cc:343] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.656124 23787 ts_tablet_manager.cc:1434] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:34.656366 23787 tablet_bootstrap.cc:492] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.656463 23471 leader_election.cc:304] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.656812 25167 consensus_queue.cc:1048] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.657230 25167 consensus_queue.cc:1048] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.657575 25167 raft_consensus.cc:3060] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.658877 23787 tablet_bootstrap.cc:654] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.660058 25235 raft_consensus.cc:515] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.660692 25235 leader_election.cc:290] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.663208 25359 raft_consensus.cc:493] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.663311 25359 raft_consensus.cc:515] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.663822 25359 leader_election.cc:290] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:34.664263 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "370a790a053446b2b06314c7db242270" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.664723 23604 leader_election.cc:343] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.666667 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d62dbf044acd4459b26d2e7e50f6206e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:34.666817 23664 raft_consensus.cc:3060] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.667840 23786 tablet_bootstrap.cc:654] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:34.667923 23333 leader_election.cc:343] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e194d03a6aa946fbb2c79d91b196440b
I20251023 20:15:34.672040 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "370a790a053446b2b06314c7db242270" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.673014 23605 leader_election.cc:343] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.673152 23605 leader_election.cc:304] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.673379 25262 raft_consensus.cc:515] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.673508 25276 raft_consensus.cc:2749] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.674001 25262 leader_election.cc:290] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.676885 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:34.677048 23662 raft_consensus.cc:3060] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.677639 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:34.678532 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:34.678987 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:34.679348 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:34.684834 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d62dbf044acd4459b26d2e7e50f6206e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:34.685012 23536 raft_consensus.cc:3060] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.686930 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e194d03a6aa946fbb2c79d91b196440b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.687139 23531 raft_consensus.cc:2468] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:34.692749 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:34.697063 25276 raft_consensus.cc:493] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.697260 25276 raft_consensus.cc:515] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.697310 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:34.697512 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:34.697855 25276 leader_election.cc:290] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.698870 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e343a7d39cf4329b99c9f9ffd8cefb2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.699421 23605 leader_election.cc:343] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.702379 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e343a7d39cf4329b99c9f9ffd8cefb2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.703306 23604 leader_election.cc:343] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.703415 23604 leader_election.cc:304] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.703830 25276 raft_consensus.cc:2749] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.706897 25454 raft_consensus.cc:493] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.707078 25454 raft_consensus.cc:515] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.707370 25262 raft_consensus.cc:493] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.707481 25262 raft_consensus.cc:515] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.708045 25262 leader_election.cc:290] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.708705 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76380b341a304d6ba31648fbd1548f05" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.708871 23666 raft_consensus.cc:2468] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.709127 25454 leader_election.cc:290] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.709427 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.709606 23666 raft_consensus.cc:2468] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.709893 23470 leader_election.cc:304] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:34.710355 23470 leader_election.cc:304] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:34.710613 25262 raft_consensus.cc:2804] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.710673 25262 raft_consensus.cc:493] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.710727 25262 raft_consensus.cc:3060] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.711964 25454 raft_consensus.cc:2804] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.712082 25454 raft_consensus.cc:493] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.712177 25454 raft_consensus.cc:3060] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.716571 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76380b341a304d6ba31648fbd1548f05" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.716650 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.717250 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:34.722283 23790 tablet_bootstrap.cc:492] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:34.722421 23790 ts_tablet_manager.cc:1403] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.002s
I20251023 20:15:34.728344 23333 leader_election.cc:304] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:34.728632 25236 raft_consensus.cc:2804] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.728704 25236 raft_consensus.cc:493] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.728758 25236 raft_consensus.cc:3060] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:34.729997 23471 leader_election.cc:343] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:34.730434 23471 leader_election.cc:343] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:34.730737 23471 leader_election.cc:343] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.731360 23790 raft_consensus.cc:359] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.731536 23790 raft_consensus.cc:385] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.731593 23790 raft_consensus.cc:740] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.731779 23790 consensus_queue.cc:260] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.732462 23790 ts_tablet_manager.cc:1434] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251023 20:15:34.732748 23790 tablet_bootstrap.cc:492] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:34.734222 23790 tablet_bootstrap.cc:654] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.737385 23717 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:34.741936 23583 heartbeater.cc:344] Connected to a master server at 127.22.127.189:33509
I20251023 20:15:34.743536 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:34.743945 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:34.744184 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:34.744405 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
I20251023 20:15:34.750763 25464 raft_consensus.cc:493] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.750923 25464 raft_consensus.cc:515] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.752192 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29c2c976d6a3440394f478a31abc7134" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.752422 23668 raft_consensus.cc:2468] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.752590 23664 raft_consensus.cc:2468] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:34.752835 23470 leader_election.cc:304] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:34.755134 23662 raft_consensus.cc:2468] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:34.756004 25282 ts_tablet_manager.cc:1929] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:34.756137 25282 log.cc:1199] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/9a6a2d2b25824dbdafaf2a9cabb0f061
I20251023 20:15:34.756462 25282 ts_tablet_manager.cc:1950] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:34.756911 25466 consensus_queue.cc:1048] T 9b766ff2dff34a8f80f70591dba54017 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.757302 25466 raft_consensus.cc:2804] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.757385 25466 raft_consensus.cc:493] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.757299 24297 raft_consensus.cc:515] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.757452 25466 raft_consensus.cc:3060] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.757988 24297 leader_election.cc:290] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.758594 24830 raft_consensus.cc:2749] T 67e0701159c548c49e9e245820c45028 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.761194 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6f3abcaa81c4646bc1432bcbd1d4fcf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:34.761359 23668 raft_consensus.cc:3060] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.750872 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:34.763953 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:34.764225 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:34.764496 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:34.764580 23333 leader_election.cc:304] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:34.764868 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:34.765033 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:34.765096 25235 raft_consensus.cc:2804] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:34.765214 25235 raft_consensus.cc:697] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:34.765456 25235 consensus_queue.cc:237] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.765641 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:34.767791 25276 raft_consensus.cc:493] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.767938 25276 raft_consensus.cc:515] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.768515 25276 leader_election.cc:290] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:34.768697 25276 raft_consensus.cc:493] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.768797 25276 raft_consensus.cc:515] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.769301 25276 leader_election.cc:290] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.770036 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a6a3425a49403e99c8ade114166e46" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.770346 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa59141039de402e9ccce427604ecb31" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.767768 23787 tablet_bootstrap.cc:492] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:34.770602 23605 leader_election.cc:343] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.770700 23787 ts_tablet_manager.cc:1403] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.114s user 0.008s sys 0.040s
W20251023 20:15:34.771075 23605 leader_election.cc:343] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.771384 23787 raft_consensus.cc:359] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.771528 23787 raft_consensus.cc:385] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.771569 23787 raft_consensus.cc:740] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.771746 23787 consensus_queue.cc:260] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.772217 23787 ts_tablet_manager.cc:1434] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:34.772469 23787 tablet_bootstrap.cc:492] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.773375 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:34.773625 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:34.773965 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:34.774134 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:34.774257 23787 tablet_bootstrap.cc:654] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.774330 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:34.774528 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:34.779371 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6f3abcaa81c4646bc1432bcbd1d4fcf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:34.779650 23531 raft_consensus.cc:3060] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.774909 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:34.780462 23219 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:34.781314 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29c2c976d6a3440394f478a31abc7134" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.781507 23398 raft_consensus.cc:2468] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.794625 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa59141039de402e9ccce427604ecb31" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.794775 23470 leader_election.cc:304] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.795032 25470 raft_consensus.cc:2804] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:34.795186 25470 raft_consensus.cc:697] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
W20251023 20:15:34.795251 23604 leader_election.cc:343] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.795367 23604 leader_election.cc:304] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.795734 25276 raft_consensus.cc:2749] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.796402 25464 leader_election.cc:290] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.795010 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a6a3425a49403e99c8ade114166e46" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:34.797204 25470 consensus_queue.cc:237] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:34.797587 23604 leader_election.cc:343] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.797694 23604 leader_election.cc:304] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.798071 23786 tablet_bootstrap.cc:492] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.798080 25359 raft_consensus.cc:2749] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.798190 23786 ts_tablet_manager.cc:1403] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.151s user 0.019s sys 0.043s
I20251023 20:15:34.798621 25040 consensus_queue.cc:1048] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.799010 23786 raft_consensus.cc:359] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.799217 23786 raft_consensus.cc:385] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.799311 23786 raft_consensus.cc:740] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.799499 23790 tablet_bootstrap.cc:492] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:34.799630 23790 ts_tablet_manager.cc:1403] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.000s
I20251023 20:15:34.799621 23786 consensus_queue.cc:260] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.800174 23786 ts_tablet_manager.cc:1434] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:34.800541 23786 tablet_bootstrap.cc:492] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.802371 23786 tablet_bootstrap.cc:654] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.803429 23790 raft_consensus.cc:359] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.803596 23790 raft_consensus.cc:385] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.803642 23790 raft_consensus.cc:740] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.809932 23790 consensus_queue.cc:260] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.810827 23790 ts_tablet_manager.cc:1434] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251023 20:15:34.810938 25172 consensus_queue.cc:1048] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.811131 23790 tablet_bootstrap.cc:492] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:34.812726 23787 tablet_bootstrap.cc:492] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.812925 23787 ts_tablet_manager.cc:1403] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.010s
I20251023 20:15:34.815987 25464 raft_consensus.cc:493] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.816162 25464 raft_consensus.cc:515] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.817292 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca11ddd5d97544cf916b87fd3dff219d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.817731 23471 leader_election.cc:343] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.816843 25464 leader_election.cc:290] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.817960 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca11ddd5d97544cf916b87fd3dff219d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:34.818136 23664 raft_consensus.cc:2468] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:34.818418 23787 raft_consensus.cc:359] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.818493 23470 leader_election.cc:304] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.818650 23787 raft_consensus.cc:385] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.818717 23787 raft_consensus.cc:740] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.818863 25464 raft_consensus.cc:2804] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:34.818977 25464 raft_consensus.cc:493] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:34.819047 25464 raft_consensus.cc:3060] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.818965 23787 consensus_queue.cc:260] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.819622 23787 ts_tablet_manager.cc:1434] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251023 20:15:34.819898 23787 tablet_bootstrap.cc:492] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.820629 23790 tablet_bootstrap.cc:654] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.821573 23787 tablet_bootstrap.cc:654] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.822966 23786 tablet_bootstrap.cc:492] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.823137 23786 ts_tablet_manager.cc:1403] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20251023 20:15:34.823915 23786 raft_consensus.cc:359] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.824123 23786 raft_consensus.cc:385] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.824195 23786 raft_consensus.cc:740] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.824453 23786 consensus_queue.cc:260] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.825022 23786 ts_tablet_manager.cc:1434] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:34.825202 25276 raft_consensus.cc:493] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.825312 25276 raft_consensus.cc:515] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.825415 23786 tablet_bootstrap.cc:492] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.825855 25276 leader_election.cc:290] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.826550 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d22aa40e5844d78a8679a3ad1b08da9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.826771 23786 tablet_bootstrap.cc:654] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:34.827042 23605 leader_election.cc:343] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.830402 25462 consensus_queue.cc:1048] T 94cd8880a78c4b4eb0c19ba5944b12f2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.830839 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d22aa40e5844d78a8679a3ad1b08da9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.831393 23604 leader_election.cc:343] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.831494 23604 leader_election.cc:304] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.831667 25276 raft_consensus.cc:2749] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.835898 23786 tablet_bootstrap.cc:492] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.836473 23786 ts_tablet_manager.cc:1403] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20251023 20:15:34.836809 23790 tablet_bootstrap.cc:492] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:34.836951 23790 ts_tablet_manager.cc:1403] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20251023 20:15:34.837623 23786 raft_consensus.cc:359] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.837625 23790 raft_consensus.cc:359] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.837800 23790 raft_consensus.cc:385] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.837852 23790 raft_consensus.cc:740] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.838065 23790 consensus_queue.cc:260] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:34.838384 23206 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5d801ca6ce63440abefe0c694889fa9e consensus state has no leader
I20251023 20:15:34.838915 23786 raft_consensus.cc:385] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.839143 23786 raft_consensus.cc:740] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.839454 23786 consensus_queue.cc:260] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.840250 23786 ts_tablet_manager.cc:1434] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:34.840947 23786 tablet_bootstrap.cc:492] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:34.842677 23786 tablet_bootstrap.cc:654] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.850729 23668 raft_consensus.cc:2468] T c6f3abcaa81c4646bc1432bcbd1d4fcf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:34.855089 25167 raft_consensus.cc:2749] T e18c1df87ee640148f5e827ee168fea6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.855531 24780 ts_tablet_manager.cc:1929] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:34.855649 24780 log.cc:1199] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/994a555b24e4432a84601c54d422aa5d
I20251023 20:15:34.855955 24780 ts_tablet_manager.cc:1950] T 994a555b24e4432a84601c54d422aa5d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:34.856178 25236 raft_consensus.cc:515] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.856801 25236 leader_election.cc:290] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.857775 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 363) took 1332 ms. Trace:
I20251023 20:15:34.857893 25282 rpcz_store.cc:276] 1023 20:15:33.525162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.525389 (+ 227us) service_pool.cc:225] Handling call
1023 20:15:34.465112 (+939723us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:34.465124 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:34.656903 (+191779us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:34.755885 (+ 98982us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:34.857753 (+101868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933793}
I20251023 20:15:34.858269 25282 tablet_replica.cc:333] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:34.858435 25282 raft_consensus.cc:2243] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:34.858565 25282 raft_consensus.cc:2272] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.862632 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 (table default.Table_cba504e93e4a4dd3b9f6c2a5e74867c9 [id=d05789186efe404f94adbf2a543ba4b1]) successfully deleted
I20251023 20:15:34.863338 25167 raft_consensus.cc:493] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.863471 25167 raft_consensus.cc:515] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.864071 25167 leader_election.cc:290] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.865682 25282 ts_tablet_manager.cc:1916] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:34.868115 23470 leader_election.cc:343] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 01fa5f9c700b458ea147373d78551573
I20251023 20:15:34.868139 25276 raft_consensus.cc:493] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.868280 25276 raft_consensus.cc:515] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.868886 25276 leader_election.cc:290] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:34.869712 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:34.870302 23605 leader_election.cc:343] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.874858 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:34.875700 23604 leader_election.cc:343] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.875845 23604 leader_election.cc:304] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:34.876219 25276 raft_consensus.cc:2749] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.884720 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:34.884950 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:34.885203 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:34.885442 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:34.885661 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:34.885874 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:34.886428 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:34.886631 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:34.887090 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:34.887279 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:34.887710 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:34.887893 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:34.888051 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:34.888216 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b680b1cbc103403b91a1104248ecbdcf (table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]). Expected version 2 got 0
I20251023 20:15:34.888370 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:34.901190 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:34.901608 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
W20251023 20:15:34.902043 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 63521.9us
I20251023 20:15:34.903117 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:34.903498 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e194d03a6aa946fbb2c79d91b196440b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:34.903846 23532 raft_consensus.cc:3060] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.904685 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:34.905306 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:34.909127 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:34.909687 23790 ts_tablet_manager.cc:1434] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.073s user 0.002s sys 0.000s
I20251023 20:15:34.912130 23790 tablet_bootstrap.cc:492] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:34.912444 23787 tablet_bootstrap.cc:492] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:34.912530 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:34.912699 23787 ts_tablet_manager.cc:1403] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.093s user 0.019s sys 0.020s
I20251023 20:15:34.912875 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:34.913264 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:34.913579 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:34.913765 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:34.913933 23787 raft_consensus.cc:359] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.914176 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:34.914182 23787 raft_consensus.cc:385] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.914304 23787 raft_consensus.cc:740] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.914615 23787 consensus_queue.cc:260] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.914744 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:34.914886 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:34.915555 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:34.915880 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:34.916129 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:34.916414 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:34.916740 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:34.917249 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:34.917402 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:34.918409 23787 ts_tablet_manager.cc:1434] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251023 20:15:34.918830 23787 tablet_bootstrap.cc:492] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:34.920563 23787 tablet_bootstrap.cc:654] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.920612 23790 tablet_bootstrap.cc:654] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:34.929435 25470 raft_consensus.cc:493] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:34.929586 25470 raft_consensus.cc:515] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.930172 25470 leader_election.cc:290] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:34.930863 23470 leader_election.cc:343] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 40c659c273c449939d0dd708d39b393f
I20251023 20:15:34.917870 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:34.934194 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:34.934592 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:34.934872 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:34.935632 23213 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:34.943053 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01fa5f9c700b458ea147373d78551573" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.943279 23333 leader_election.cc:304] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:34.943434 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40c659c273c449939d0dd708d39b393f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:34.943655 25172 raft_consensus.cc:2804] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:34.943771 25172 raft_consensus.cc:697] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:34.944022 25172 consensus_queue.cc:237] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.945739 23218 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 00ec606cbf834ccd8762e034b520e58e
I20251023 20:15:34.946374 23218 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 77) took 6736 ms. Trace:
I20251023 20:15:34.946508 23218 rpcz_store.cc:276] 1023 20:15:28.210039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.877071 (+3667032us) service_pool.cc:225] Handling call
1023 20:15:31.877584 (+ 513us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:31.877722 (+ 138us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:31.877788 (+ 66us) catalog_manager.cc:2824] Locking tablets
1023 20:15:33.617319 (+1739531us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:33.644674 (+ 27355us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:33.701132 (+ 56458us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:34.945654 (+1244522us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:34.946325 (+ 671us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:33.621175 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:33.621304 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:33.621311 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:33.621313 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:33.621437 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:33.621450 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:33.621455 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:33.621564 (+ 109us) tablet.cc:699] Row locks acquired
1023 20:15:33.621567 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:33.621608 (+ 41us) write_op.cc:270] Start()
1023 20:15:33.621644 (+ 36us) write_op.cc:276] Timestamp: P: 1761250533620144 usec, L: 0
1023 20:15:33.621648 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:33.630899 (+ 9251us) log.cc:844] Serialized 768 byte log entry
1023 20:15:33.631098 (+ 199us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 000000000098330a 00007f899878e682 00007f8999b70ca5 00007f8999b6f7c9 00007f8999b77bf0 00007f8999b779fd 0000000000a880ec 00007f898c95576f 00007f898c958a3b 00007f898c9588b8 0000000000a880ec
1023 20:15:33.641167 (+ 10069us) op_driver.cc:464] REPLICATION: finished
1023 20:15:33.643768 (+ 2601us) write_op.cc:301] APPLY: starting
1023 20:15:33.643785 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:33.643829 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:33.643832 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:33.643916 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:33.643949 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:33.643957 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:33.644034 (+ 77us) log.cc:844] Serialized 29 byte log entry
1023 20:15:33.644097 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:33.644117 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:33.644123 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:33.644975 (+ 852us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:33.644981 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2530,"apply.run_cpu_time_us":452,"apply.run_wall_time_us":981,"num_ops":2,"prepare.queue_time_us":3110,"prepare.run_cpu_time_us":1072,"prepare.run_wall_time_us":10006,"raft.queue_time_us":847,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":359,"replication_time_us":19484,"spinlock_wait_cycles":20697984}]]}
I20251023 20:15:34.950428 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 00ec606cbf834ccd8762e034b520e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:34.950886 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 00ec606cbf834ccd8762e034b520e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:34.952845 23333 leader_election.cc:343] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: e194d03a6aa946fbb2c79d91b196440b
I20251023 20:15:34.953887 23536 raft_consensus.cc:2468] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:34.954349 24821 raft_consensus.cc:515] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.954942 24821 leader_election.cc:290] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:34.961163 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 164) took 6187 ms. Trace:
I20251023 20:15:34.961256 24780 rpcz_store.cc:276] 1023 20:15:28.773831 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.774049 (+ 218us) service_pool.cc:225] Handling call
1023 20:15:34.260908 (+5486859us) ts_tablet_manager.cc:1151] Deleting tablet 994a555b24e4432a84601c54d422aa5d
1023 20:15:34.260919 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:34.459490 (+198571us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:34.855417 (+395927us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:34.961136 (+105719us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57,"tablet-delete.queue_time_us":5486700}
I20251023 20:15:34.961601 24780 tablet_replica.cc:333] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:34.961788 24780 raft_consensus.cc:2243] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:34.962699 24780 raft_consensus.cc:2272] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:34.963277 24780 ts_tablet_manager.cc:1916] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:34.965984 23471 leader_election.cc:343] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.966115 23471 leader_election.cc:304] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
W20251023 20:15:34.966683 23471 leader_election.cc:343] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:34.966720 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 994a555b24e4432a84601c54d422aa5d (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]) successfully deleted
I20251023 20:15:34.966782 23471 leader_election.cc:304] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:34.966987 25167 raft_consensus.cc:2749] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.967128 25167 consensus_queue.cc:1048] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:34.967453 25462 raft_consensus.cc:2749] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:34.967691 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e36bb5e61d34b35a373bd6a51a7ac5e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:34.967837 23534 raft_consensus.cc:3060] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:34.968346 23362 tablet_service.cc:1552] Processing DeleteTablet for tablet 00ec606cbf834ccd8762e034b520e58e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:34.984114 23333 leader_election.cc:343] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 7e36bb5e61d34b35a373bd6a51a7ac5e
I20251023 20:15:34.995874 23786 tablet_bootstrap.cc:492] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:34.996107 23786 ts_tablet_manager.cc:1403] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.155s user 0.025s sys 0.037s
I20251023 20:15:34.996881 23786 raft_consensus.cc:359] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.997166 23786 raft_consensus.cc:385] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:34.997262 23786 raft_consensus.cc:740] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:34.997550 23786 consensus_queue.cc:260] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:34.998080 23786 ts_tablet_manager.cc:1434] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:34.998427 23786 tablet_bootstrap.cc:492] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.008374 25470 consensus_queue.cc:1048] T f03df77c344b4736a30725e4d9f72a35 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.010550 24821 consensus_queue.cc:1048] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.012053 25420 mvcc.cc:204] Tried to move back new op lower bound from 7214082163176202240 to 7214082143784493056. Current Snapshot: MvccSnapshot[applied={T|T < 7214082161080590336}]
I20251023 20:15:35.017042 23790 tablet_bootstrap.cc:492] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.017278 23790 ts_tablet_manager.cc:1403] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.105s user 0.003s sys 0.000s
I20251023 20:15:35.018092 23790 raft_consensus.cc:359] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.018319 23790 raft_consensus.cc:385] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.018396 23790 raft_consensus.cc:740] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.018627 23790 consensus_queue.cc:260] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.019290 23790 ts_tablet_manager.cc:1434] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.019616 23790 tablet_bootstrap.cc:492] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.022055 23787 tablet_bootstrap.cc:492] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.022245 23787 ts_tablet_manager.cc:1403] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.103s user 0.026s sys 0.028s
I20251023 20:15:35.022655 23790 tablet_bootstrap.cc:654] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.023039 23787 raft_consensus.cc:359] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.023262 23787 raft_consensus.cc:385] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.023356 23787 raft_consensus.cc:740] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.023633 23787 consensus_queue.cc:260] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.024447 23787 ts_tablet_manager.cc:1434] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:35.024778 23787 tablet_bootstrap.cc:492] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.026814 23787 tablet_bootstrap.cc:654] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.032189 23787 tablet_bootstrap.cc:492] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.032382 23787 ts_tablet_manager.cc:1403] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20251023 20:15:35.033229 23787 raft_consensus.cc:359] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.033445 23787 raft_consensus.cc:385] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.033627 23787 raft_consensus.cc:740] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.033888 23787 consensus_queue.cc:260] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.034494 23787 ts_tablet_manager.cc:1434] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.034813 23787 tablet_bootstrap.cc:492] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.035140 23790 tablet_bootstrap.cc:492] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.035305 23790 ts_tablet_manager.cc:1403] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20251023 20:15:35.036077 23790 raft_consensus.cc:359] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.036275 23790 raft_consensus.cc:385] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.036361 23790 raft_consensus.cc:740] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.036671 23787 tablet_bootstrap.cc:654] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.036608 23790 consensus_queue.cc:260] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.040155 23790 ts_tablet_manager.cc:1434] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:35.040486 23790 tablet_bootstrap.cc:492] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.042425 23790 tablet_bootstrap.cc:654] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.044834 23787 tablet_bootstrap.cc:492] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.045022 23787 ts_tablet_manager.cc:1403] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20251023 20:15:35.045874 23787 raft_consensus.cc:359] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.046113 23787 raft_consensus.cc:385] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.046200 23787 raft_consensus.cc:740] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.046452 23787 consensus_queue.cc:260] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.047107 23787 ts_tablet_manager.cc:1434] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:35.047420 23787 tablet_bootstrap.cc:492] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.049259 23787 tablet_bootstrap.cc:654] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.051667 23790 tablet_bootstrap.cc:492] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.051857 23790 ts_tablet_manager.cc:1403] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20251023 20:15:35.052666 23790 raft_consensus.cc:359] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.052874 23790 raft_consensus.cc:385] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.052953 23790 raft_consensus.cc:740] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.053200 23790 consensus_queue.cc:260] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.054946 23790 ts_tablet_manager.cc:1434] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:35.055274 23790 tablet_bootstrap.cc:492] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.055298 25282 ts_tablet_manager.cc:1929] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:35.055418 25282 log.cc:1199] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/f9bd2b109bfa4804a372965e60ba6e85
I20251023 20:15:35.055716 25282 ts_tablet_manager.cc:1950] T f9bd2b109bfa4804a372965e60ba6e85 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:35.056078 25454 raft_consensus.cc:515] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.056752 25454 leader_election.cc:290] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.057406 23787 tablet_bootstrap.cc:492] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.057515 23787 ts_tablet_manager.cc:1403] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20251023 20:15:35.058185 23787 raft_consensus.cc:359] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.058326 23787 raft_consensus.cc:385] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.058368 23787 raft_consensus.cc:740] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.058581 23787 consensus_queue.cc:260] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.062285 23787 ts_tablet_manager.cc:1434] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251023 20:15:35.062546 23787 tablet_bootstrap.cc:492] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.064230 23787 tablet_bootstrap.cc:654] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.068221 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:35.069005 23790 tablet_bootstrap.cc:654] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.074510 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:35.074769 23665 raft_consensus.cc:3060] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.077232 23787 tablet_bootstrap.cc:492] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.077481 25492 tablet_replica.cc:333] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:35.077569 23787 ts_tablet_manager.cc:1403] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20251023 20:15:35.077713 25492 raft_consensus.cc:2243] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.077881 25492 raft_consensus.cc:2272] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.078280 23787 raft_consensus.cc:359] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.078430 23787 raft_consensus.cc:385] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.078444 25492 ts_tablet_manager.cc:1916] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:35.078478 23787 raft_consensus.cc:740] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.078680 23787 consensus_queue.cc:260] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.079277 23787 ts_tablet_manager.cc:1434] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.079531 23787 tablet_bootstrap.cc:492] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.079538 24821 raft_consensus.cc:493] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.079654 24821 raft_consensus.cc:515] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.080227 24821 leader_election.cc:290] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.081202 23787 tablet_bootstrap.cc:654] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.082001 23790 tablet_bootstrap.cc:492] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.082127 23790 ts_tablet_manager.cc:1403] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.002s
I20251023 20:15:35.082543 24821 raft_consensus.cc:493] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.082652 24821 raft_consensus.cc:515] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.083158 24821 leader_election.cc:290] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.083846 23790 raft_consensus.cc:359] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.084010 23790 raft_consensus.cc:385] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.084055 23790 raft_consensus.cc:740] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.084254 23790 consensus_queue.cc:260] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:35.087277 23471 leader_election.cc:343] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.087745 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.088773 23790 ts_tablet_manager.cc:1434] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:35.089097 23790 tablet_bootstrap.cc:492] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.089207 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72408899f710470baaf6fdb268cb04c8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.089372 23533 raft_consensus.cc:2468] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:35.089988 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1f9935f5e4f4daa9df8fe1611ed76c9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.090195 23663 raft_consensus.cc:2393] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:35.090730 23790 tablet_bootstrap.cc:654] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.093617 23787 tablet_bootstrap.cc:492] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.093818 23787 ts_tablet_manager.cc:1403] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20251023 20:15:35.094096 23333 leader_election.cc:304] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.094530 23333 leader_election.cc:304] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.094751 23787 raft_consensus.cc:359] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:35.094894 23333 leader_election.cc:343] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 72408899f710470baaf6fdb268cb04c8
I20251023 20:15:35.095052 23787 raft_consensus.cc:385] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.095156 23787 raft_consensus.cc:740] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.095392 25489 raft_consensus.cc:2804] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251023 20:15:35.095470 25489 raft_consensus.cc:493] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.095537 25489 raft_consensus.cc:3060] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:35.095419 23787 consensus_queue.cc:260] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.096143 23787 ts_tablet_manager.cc:1434] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.096419 23787 tablet_bootstrap.cc:492] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.096820 24925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52056 (request call id 0) took 4926 ms. Trace:
I20251023 20:15:35.097937 23787 tablet_bootstrap.cc:654] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.098091 25235 consensus_queue.cc:1048] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.098453 25454 raft_consensus.cc:493] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.098501 25235 raft_consensus.cc:3060] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.098592 25454 raft_consensus.cc:515] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.099187 25454 leader_election.cc:290] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.099880 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e658aa360e524a9f9bdb35c1f68ae426" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.100029 23666 raft_consensus.cc:2468] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.100386 23470 leader_election.cc:304] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.100747 25454 raft_consensus.cc:2804] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.100826 25454 raft_consensus.cc:493] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.100881 25454 raft_consensus.cc:3060] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.108168 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e658aa360e524a9f9bdb35c1f68ae426" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.108351 23392 raft_consensus.cc:2468] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.110191 25488 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 690ffde75cd745b6a87925967d54471a)
I20251023 20:15:35.110461 25488 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:35.121066 25167 raft_consensus.cc:493] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.103139 25469 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "690ffde75cd745b6a87925967d54471a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:35.121266 25469 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:35.121258 25167 raft_consensus.cc:515] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.121631 25359 raft_consensus.cc:493] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.121789 25359 raft_consensus.cc:515] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.121968 25167 leader_election.cc:290] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.122375 25359 leader_election.cc:290] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:35.123670 25236 raft_consensus.cc:493] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:35.123809 25236 raft_consensus.cc:515] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:35.124094 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bce63866d524067b87145b07347ffcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.124353 25236 leader_election.cc:290] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.124646 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bce63866d524067b87145b07347ffcf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:35.125458 23604 leader_election.cc:343] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.125735 23605 leader_election.cc:343] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.125831 23605 leader_election.cc:304] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.126281 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b220ceff0924b26ae8a54dfb8705456" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.126623 25359 raft_consensus.cc:2749] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:35.127831 23333 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb
W20251023 20:15:35.127897 23471 leader_election.cc:343] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.128756 23333 leader_election.cc:343] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 86b9f4c5d9e0469c87071a33722ce3eb
I20251023 20:15:35.128862 23333 leader_election.cc:304] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.128911 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b220ceff0924b26ae8a54dfb8705456" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.129005 25359 raft_consensus.cc:493] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.129155 25359 raft_consensus.cc:515] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.129329 25236 raft_consensus.cc:2749] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:35.129730 25359 leader_election.cc:290] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:35.130373 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19da816a483a4d5b9dc768ef697d54ec" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.130741 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19da816a483a4d5b9dc768ef697d54ec" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.131224 23604 leader_election.cc:343] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.132107 23605 leader_election.cc:343] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.132182 23605 leader_election.cc:304] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.133319 25359 raft_consensus.cc:2749] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.130777 23670 raft_consensus.cc:2468] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.134049 23470 leader_election.cc:304] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:35.134441 25167 raft_consensus.cc:2804] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.134516 25167 raft_consensus.cc:493] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.134572 25167 raft_consensus.cc:3060] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.134989 25399 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690ffde75cd745b6a87925967d54471a. Latest consensus state: current_term: 2 leader_uuid: "690ffde75cd745b6a87925967d54471a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:35.135277 25399 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:35.145009 25488 leader_election.cc:290] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:35.147507 25040 consensus_queue.cc:1048] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.152051 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 364) took 1626 ms. Trace:
I20251023 20:15:35.152163 25282 rpcz_store.cc:276] 1023 20:15:33.525331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.525553 (+ 222us) service_pool.cc:225] Handling call
1023 20:15:34.858223 (+1332670us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:34.858233 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:34.955212 (+ 96979us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.055179 (+ 99967us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.152028 (+ 96849us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326854}
I20251023 20:15:35.152494 25282 tablet_replica.cc:333] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:35.152705 25282 raft_consensus.cc:2243] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.152859 25282 raft_consensus.cc:2272] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.153535 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet f9bd2b109bfa4804a372965e60ba6e85 (table default.Table_a0beeea080b946158748ce5a6db61212 [id=108a2f4bdbd943139aa407a6197c1afd]) successfully deleted
I20251023 20:15:35.153694 25282 ts_tablet_manager.cc:1916] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:35.157101 23531 raft_consensus.cc:2468] T c6f3abcaa81c4646bc1432bcbd1d4fcf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:35.157617 25489 raft_consensus.cc:515] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.158360 25489 leader_election.cc:290] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.159823 24780 ts_tablet_manager.cc:1929] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:35.159929 24780 log.cc:1199] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/bb475e303d07491bbc19dc3f628bceb7
I20251023 20:15:35.160212 24780 ts_tablet_manager.cc:1950] T bb475e303d07491bbc19dc3f628bceb7 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:35.160548 23665 raft_consensus.cc:2468] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:35.164268 23470 leader_election.cc:304] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:35.165843 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72408899f710470baaf6fdb268cb04c8" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:35.166026 23531 raft_consensus.cc:3060] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Advancing to term 2
I20251023 20:15:35.165867 25470 raft_consensus.cc:2804] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.166276 25470 raft_consensus.cc:697] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:35.166531 25470 consensus_queue.cc:237] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:35.168279 23333 leader_election.cc:343] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 72408899f710470baaf6fdb268cb04c8
I20251023 20:15:35.169533 23786 tablet_bootstrap.cc:654] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.170379 25359 raft_consensus.cc:493] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.170503 25359 raft_consensus.cc:515] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.171036 25359 leader_election.cc:290] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.171763 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb7935c4cf64e2f802ca12c1f63317b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.172215 23790 tablet_bootstrap.cc:492] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.172559 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb7935c4cf64e2f802ca12c1f63317b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.172878 23790 ts_tablet_manager.cc:1403] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.008s
W20251023 20:15:35.173159 23605 leader_election.cc:343] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.173931 23604 leader_election.cc:343] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.174027 23604 leader_election.cc:304] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.174378 25359 raft_consensus.cc:2749] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.176152 25514 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251023 20:15:35.175190 23790 raft_consensus.cc:359] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.182077 23790 raft_consensus.cc:385] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.182163 23790 raft_consensus.cc:740] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.182382 23790 consensus_queue.cc:260] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.182977 23790 ts_tablet_manager.cc:1434] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:35.184551 23790 tablet_bootstrap.cc:492] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.191502 25359 raft_consensus.cc:493] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.191656 25359 raft_consensus.cc:515] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.192302 25359 leader_election.cc:290] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.193022 23790 tablet_bootstrap.cc:654] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.193406 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9be2a866cf46cfbec8db23a6d3f572" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.193589 23787 tablet_bootstrap.cc:492] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.193708 23787 ts_tablet_manager.cc:1403] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.097s user 0.016s sys 0.011s
W20251023 20:15:35.194219 23605 leader_election.cc:343] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.194525 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9be2a866cf46cfbec8db23a6d3f572" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.194559 23787 raft_consensus.cc:359] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:35.195092 23604 leader_election.cc:343] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.195256 23604 leader_election.cc:304] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.195629 25359 raft_consensus.cc:2749] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.195854 23787 raft_consensus.cc:385] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.196079 23787 raft_consensus.cc:740] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.196589 23787 consensus_queue.cc:260] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.209486 23787 ts_tablet_manager.cc:1434] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20251023 20:15:35.209921 23787 tablet_bootstrap.cc:492] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.212062 25359 raft_consensus.cc:493] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.212199 25359 raft_consensus.cc:515] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.212726 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:35.212833 25359 leader_election.cc:290] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.212981 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:35.213212 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:35.213546 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:35.213893 23787 tablet_bootstrap.cc:654] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.213824 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b446550686024c3ca80f106112a2110f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.213907 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:35.214120 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
W20251023 20:15:35.214329 23605 leader_election.cc:343] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.214331 25359 raft_consensus.cc:493] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.214541 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b446550686024c3ca80f106112a2110f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.214597 25359 raft_consensus.cc:515] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.214761 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
W20251023 20:15:35.215029 23604 leader_election.cc:343] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.215114 25359 leader_election.cc:290] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.215130 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]). Expected version 1 got 0
I20251023 20:15:35.215124 23604 leader_election.cc:304] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.215276 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]). Expected version 1 got 0
I20251023 20:15:35.215535 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:35.215592 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59598ea40b9e4e5da0ec36f35f1094d6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.215700 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:35.215704 25359 raft_consensus.cc:2749] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.215837 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:35.216048 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:35.216223 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59598ea40b9e4e5da0ec36f35f1094d6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:35.216279 23605 leader_election.cc:343] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.216336 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:35.216518 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
W20251023 20:15:35.216692 23604 leader_election.cc:343] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.216809 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:35.216794 23604 leader_election.cc:304] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.217129 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:35.217239 25359 raft_consensus.cc:2749] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.217650 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:35.217818 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:35.220724 23786 tablet_bootstrap.cc:492] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.220919 23786 ts_tablet_manager.cc:1403] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.223s user 0.012s sys 0.001s
I20251023 20:15:35.221688 23786 raft_consensus.cc:359] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.221891 23786 raft_consensus.cc:385] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.221998 23786 raft_consensus.cc:740] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.222260 23786 consensus_queue.cc:260] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.222827 23786 ts_tablet_manager.cc:1434] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.223155 23786 tablet_bootstrap.cc:492] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.224680 24297 raft_consensus.cc:493] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.224862 24297 raft_consensus.cc:515] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.225317 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:35.225512 24297 leader_election.cc:290] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.225662 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:35.226214 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:35.226415 24297 raft_consensus.cc:493] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.226588 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:35.226549 24297 raft_consensus.cc:515] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:35.226742 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92576756ec574ceaa4c81f435cb72301" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.226816 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]). Expected version 2 got 0
I20251023 20:15:35.226933 23665 raft_consensus.cc:2468] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.227077 24297 leader_election.cc:290] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:35.227483 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:35.227486 23333 leader_election.cc:304] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:35.227787 24297 raft_consensus.cc:2804] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.227782 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:35.227855 24297 raft_consensus.cc:493] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.227912 24297 raft_consensus.cc:3060] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.228001 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:35.228248 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
W20251023 20:15:35.228229 23333 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:35.228302 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92576756ec574ceaa4c81f435cb72301" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:35.228410 23333 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:35.228469 23536 raft_consensus.cc:2468] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.228478 23333 leader_election.cc:304] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.228559 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:35.228752 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:35.229008 25172 raft_consensus.cc:2749] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:35.229063 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]). Expected version 3 got 0
I20251023 20:15:35.229342 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:35.229779 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:35.230331 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:35.230721 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:35.230926 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:35.231215 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:35.231401 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:35.231611 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:35.242305 24902 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9bc6522db11e4960a9aad84435371a32" instance_seqno: 1761250514256798) as {username='slave'} at 127.22.127.131:33585; Asking this server to re-register.
I20251023 20:15:35.245007 23717 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:35.245059 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]). Expected version 1 got 0
I20251023 20:15:35.251088 23790 tablet_bootstrap.cc:492] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.251319 23790 ts_tablet_manager.cc:1403] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.067s user 0.014s sys 0.002s
I20251023 20:15:35.252038 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 165) took 6477 ms. Trace:
I20251023 20:15:35.252130 24780 rpcz_store.cc:276] 1023 20:15:28.774355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.774503 (+ 148us) service_pool.cc:225] Handling call
1023 20:15:34.961551 (+6187048us) ts_tablet_manager.cc:1151] Deleting tablet bb475e303d07491bbc19dc3f628bceb7
1023 20:15:34.961561 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.057057 (+ 95496us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.159696 (+102639us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.252019 (+ 92323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6186866}
I20251023 20:15:35.252619 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 166) took 6476 ms. Trace:
I20251023 20:15:35.252744 25235 raft_consensus.cc:2749] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.252689 24780 rpcz_store.cc:276] 1023 20:15:28.776232 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.776466 (+ 234us) service_pool.cc:225] Handling call
1023 20:15:35.252480 (+6476014us) ts_tablet_manager.cc:1151] Deleting tablet fbab01ecb27a426188689408079eaa01
1023 20:15:35.252489 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.252585 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6475797}
I20251023 20:15:35.253098 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 167) took 6476 ms. Trace:
I20251023 20:15:35.253181 24780 rpcz_store.cc:276] 1023 20:15:28.776852 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.777011 (+ 159us) service_pool.cc:225] Handling call
1023 20:15:35.252972 (+6475961us) ts_tablet_manager.cc:1151] Deleting tablet ea0cbd6bec674cf5831486f500cc7b36
1023 20:15:35.252981 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.253059 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6475764}
I20251023 20:15:35.253389 24780 tablet_replica.cc:333] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:35.253552 24780 raft_consensus.cc:2243] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:35.253855 24780 raft_consensus.cc:2272] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.254359 24780 ts_tablet_manager.cc:1916] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:35.255172 24297 raft_consensus.cc:515] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.256306 24297 leader_election.cc:290] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.257813 24902 ts_manager.cc:194] Registered new tserver with Master: 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.258709 25464 raft_consensus.cc:515] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.259313 25464 leader_election.cc:290] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.252174 23790 raft_consensus.cc:359] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.259953 23790 raft_consensus.cc:385] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.260007 23790 raft_consensus.cc:740] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.260226 23790 consensus_queue.cc:260] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.260532 25492 ts_tablet_manager.cc:1929] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:35.260638 25492 log.cc:1199] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/00ec606cbf834ccd8762e034b520e58e
I20251023 20:15:35.260718 23790 ts_tablet_manager.cc:1434] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.009s user 0.000s sys 0.000s
I20251023 20:15:35.260964 25492 ts_tablet_manager.cc:1950] T 00ec606cbf834ccd8762e034b520e58e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:35.261003 23790 tablet_bootstrap.cc:492] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.262696 23790 tablet_bootstrap.cc:654] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.262733 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca11ddd5d97544cf916b87fd3dff219d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:35.262873 23392 raft_consensus.cc:3060] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.264720 23786 tablet_bootstrap.cc:654] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.265616 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92576756ec574ceaa4c81f435cb72301" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:35.265815 23665 raft_consensus.cc:3060] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.268227 23787 tablet_bootstrap.cc:492] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.268361 23787 ts_tablet_manager.cc:1403] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20251023 20:15:35.268306 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet bb475e303d07491bbc19dc3f628bceb7 (table default.Table_0d31043ac68d4a0394d7ba5b60d539c8 [id=0a9d3a3303d945f199d265637a859772]) successfully deleted
W20251023 20:15:35.268641 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet fbab01ecb27a426188689408079eaa01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbab01ecb27a426188689408079eaa01 already in progress: creating tablet
W20251023 20:15:35.268877 23205 catalog_manager.cc:4965] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet ea0cbd6bec674cf5831486f500cc7b36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0cbd6bec674cf5831486f500cc7b36 already in progress: creating tablet
I20251023 20:15:35.269138 23787 raft_consensus.cc:359] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.269291 23787 raft_consensus.cc:385] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.269340 23787 raft_consensus.cc:740] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.269503 23787 consensus_queue.cc:260] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.270054 23787 ts_tablet_manager.cc:1434] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:35.270196 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca11ddd5d97544cf916b87fd3dff219d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:35.270351 23664 raft_consensus.cc:3060] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.270862 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "92576756ec574ceaa4c81f435cb72301" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:35.271025 23536 raft_consensus.cc:3060] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.271112 23787 tablet_bootstrap.cc:492] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.273314 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:35.273697 23218 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:35.274637 25359 raft_consensus.cc:493] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.274772 25359 raft_consensus.cc:515] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.275348 25359 leader_election.cc:290] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.275923 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db758a7a7a89410f8ac1da72048fbc56" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.276563 23605 leader_election.cc:343] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.276926 23787 tablet_bootstrap.cc:654] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.277001 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db758a7a7a89410f8ac1da72048fbc56" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.277498 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_d22911da756748fb8939006df660e885 [id=78fe983663c94eabab93ee902bde6998] alter complete (version 5)
W20251023 20:15:35.277542 23604 leader_election.cc:343] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.277626 23604 leader_election.cc:304] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.278053 25359 raft_consensus.cc:2749] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.279067 23790 tablet_bootstrap.cc:492] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.283634 23790 ts_tablet_manager.cc:1403] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251023 20:15:35.284436 23790 raft_consensus.cc:359] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.284677 23790 raft_consensus.cc:385] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.284843 23790 raft_consensus.cc:740] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.096947 24925 rpcz_store.cc:276] 1023 20:15:30.170687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.170740 (+ 53us) service_pool.cc:225] Handling call
1023 20:15:34.299994 (+4129254us) raft_consensus.cc:1471] Updating replica for 83 ops
1023 20:15:34.384638 (+ 84644us) raft_consensus.cc:1534] Early marking committed up to index 0
1023 20:15:34.384649 (+ 11us) raft_consensus.cc:1539] Triggering prepare for 83 ops
1023 20:15:34.386551 (+ 1902us) write_op.cc:270] Start()
1023 20:15:34.386628 (+ 77us) write_op.cc:276] Timestamp: P: 1761250523999108 usec, L: 0
1023 20:15:34.421032 (+ 34404us) write_op.cc:270] Start()
1023 20:15:34.422420 (+ 1388us) write_op.cc:276] Timestamp: P: 1761250524170484 usec, L: 0
1023 20:15:34.423384 (+ 964us) write_op.cc:270] Start()
1023 20:15:34.423440 (+ 56us) write_op.cc:276] Timestamp: P: 1761250524171354 usec, L: 0
1023 20:15:34.426777 (+ 3337us) write_op.cc:270] Start()
1023 20:15:34.426818 (+ 41us) write_op.cc:276] Timestamp: P: 1761250524213636 usec, L: 0
1023 20:15:34.427453 (+ 635us) write_op.cc:270] Start()
1023 20:15:34.427480 (+ 27us) write_op.cc:276] Timestamp: P: 1761250524214747 usec, L: 0
1023 20:15:34.428029 (+ 549us) write_op.cc:270] Start()
1023 20:15:34.428052 (+ 23us) write_op.cc:276] Timestamp: P: 1761250524215736 usec, L: 0
1023 20:15:34.428615 (+ 563us) write_op.cc:270] Start()
1023 20:15:34.428667 (+ 52us) write_op.cc:276] Timestamp: P: 1761250524216568 usec, L: 0
1023 20:15:34.438299 (+ 9632us) write_op.cc:270] Start()
1023 20:15:34.438340 (+ 41us) write_op.cc:276] Timestamp: P: 1761250524217359 usec, L: 0
1023 20:15:34.439082 (+ 742us) write_op.cc:270] Start()
1023 20:15:34.439120 (+ 38us) write_op.cc:276] Timestamp: P: 1761250524433658 usec, L: 0
1023 20:15:34.440804 (+ 1684us) write_op.cc:270] Start()
1023 20:15:34.440844 (+ 40us) write_op.cc:276] Timestamp: P: 1761250524434308 usec, L: 0
1023 20:15:34.441690 (+ 846us) write_op.cc:270] Start()
1023 20:15:34.441731 (+ 41us) write_op.cc:276] Timestamp: P: 1761250524434857 usec, L: 0
1023 20:15:34.443364 (+ 1633us) write_op.cc:270] Start()
1023 20:15:34.443388 (+ 24us) write_op.cc:276] Timestamp: P: 1761250524447378 usec, L: 0
1023 20:15:34.444236 (+ 848us) write_op.cc:270] Start()
1023 20:15:34.444951 (+ 715us) write_op.cc:276] Timestamp: P: 1761250524519136 usec, L: 0
1023 20:15:34.445539 (+ 588us) write_op.cc:270] Start()
1023 20:15:34.445564 (+ 25us) write_op.cc:276] Timestamp: P: 1761250524530849 usec, L: 0
1023 20:15:34.447290 (+ 1726us) write_op.cc:270] Start()
1023 20:15:34.447322 (+ 32us) write_op.cc:276] Timestamp: P: 1761250524600582 usec, L: 0
1023 20:15:34.447825 (+ 503us) write_op.cc:270] Start()
1023 20:15:34.447848 (+ 23us) write_op.cc:276] Timestamp: P: 1761250524610933 usec, L: 0
1023 20:15:34.448332 (+ 484us) write_op.cc:270] Start()
1023 20:15:34.448368 (+ 36us) write_op.cc:276] Timestamp: P: 1761250524619338 usec, L: 0
1023 20:15:34.448981 (+ 613us) write_op.cc:270] Start()
1023 20:15:34.449005 (+ 24us) write_op.cc:276] Timestamp: P: 1761250524626806 usec, L: 0
1023 20:15:34.495365 (+ 46360us) write_op.cc:270] Start()
1023 20:15:34.495416 (+ 51us) write_op.cc:276] Timestamp: P: 1761250524665705 usec, L: 0
1023 20:15:34.511416 (+ 16000us) write_op.cc:270] Start()
1023 20:15:34.511457 (+ 41us) write_op.cc:276] Timestamp: P: 1761250524881539 usec, L: 0
1023 20:15:34.511645 (+ 188us) write_op.cc:270] Start()
1023 20:15:34.511671 (+ 26us) write_op.cc:276] Timestamp: P: 1761250524893446 usec, L: 0
1023 20:15:34.511864 (+ 193us) write_op.cc:270] Start()
1023 20:15:34.511887 (+ 23us) write_op.cc:276] Timestamp: P: 1761250524905331 usec, L: 0
1023 20:15:34.512027 (+ 140us) write_op.cc:270] Start()
1023 20:15:34.512053 (+ 26us) write_op.cc:276] Timestamp: P: 1761250524942637 usec, L: 0
1023 20:15:34.543573 (+ 31520us) write_op.cc:270] Start()
1023 20:15:34.543637 (+ 64us) write_op.cc:276] Timestamp: P: 1761250524954431 usec, L: 0
1023 20:15:34.543947 (+ 310us) write_op.cc:270] Start()
1023 20:15:34.543973 (+ 26us) write_op.cc:276] Timestamp: P: 1761250524967456 usec, L: 0
1023 20:15:34.544148 (+ 175us) write_op.cc:270] Start()
1023 20:15:34.544170 (+ 22us) write_op.cc:276] Timestamp: P: 1761250524990826 usec, L: 0
1023 20:15:34.544353 (+ 183us) write_op.cc:270] Start()
1023 20:15:34.544376 (+ 23us) write_op.cc:276] Timestamp: P: 1761250525018036 usec, L: 0
1023 20:15:34.544540 (+ 164us) write_op.cc:270] Start()
1023 20:15:34.544563 (+ 23us) write_op.cc:276] Timestamp: P: 1761250525039256 usec, L: 0
1023 20:15:34.544702 (+ 139us) write_op.cc:270] Start()
1023 20:15:34.544738 (+ 36us) write_op.cc:276] Timestamp: P: 1761250525040753 usec, L: 0
1023 20:15:34.544895 (+ 157us) write_op.cc:270] Start()
1023 20:15:34.544917 (+ 22us) write_op.cc:276] Timestamp: P: 1761250525041812 usec, L: 0
1023 20:15:34.545054 (+ 137us) write_op.cc:270] Start()
1023 20:15:34.545070 (+ 16us) write_op.cc:276] Timestamp: P: 1761250525042653 usec, L: 0
1023 20:15:34.572870 (+ 27800us) write_op.cc:270] Start()
1023 20:15:34.572915 (+ 45us) write_op.cc:276] Timestamp: P: 1761250525089245 usec, L: 0
1023 20:15:34.584103 (+ 11188us) write_op.cc:270] Start()
1023 20:15:34.598576 (+ 14473us) write_op.cc:276] Timestamp: P: 1761250525302300 usec, L: 0
1023 20:15:34.603611 (+ 5035us) write_op.cc:270] Start()
1023 20:15:34.603672 (+ 61us) write_op.cc:276] Timestamp: P: 1761250525302953 usec, L: 0
1023 20:15:34.604278 (+ 606us) write_op.cc:270] Start()
1023 20:15:34.604320 (+ 42us) write_op.cc:276] Timestamp: P: 1761250525420870 usec, L: 0
1023 20:15:34.604485 (+ 165us) write_op.cc:270] Start()
1023 20:15:34.604504 (+ 19us) write_op.cc:276] Timestamp: P: 1761250525421904 usec, L: 0
1023 20:15:34.604642 (+ 138us) write_op.cc:270] Start()
1023 20:15:34.604672 (+ 30us) write_op.cc:276] Timestamp: P: 1761250525510940 usec, L: 0
1023 20:15:34.604813 (+ 141us) write_op.cc:270] Start()
1023 20:15:34.604831 (+ 18us) write_op.cc:276] Timestamp: P: 1761250525522911 usec, L: 0
1023 20:15:34.604963 (+ 132us) write_op.cc:270] Start()
1023 20:15:34.604981 (+ 18us) write_op.cc:276] Timestamp: P: 1761250525557480 usec, L: 0
1023 20:15:34.605113 (+ 132us) write_op.cc:270] Start()
1023 20:15:34.605132 (+ 19us) write_op.cc:276] Timestamp: P: 1761250525787937 usec, L: 0
1023 20:15:34.605258 (+ 126us) write_op.cc:270] Start()
1023 20:15:34.605274 (+ 16us) write_op.cc:276] Timestamp: P: 1761250525798031 usec, L: 0
1023 20:15:34.605386 (+ 112us) write_op.cc:270] Start()
1023 20:15:34.605402 (+ 16us) write_op.cc:276] Timestamp: P: 1761250525809777 usec, L: 0
1023 20:15:34.605517 (+ 115us) write_op.cc:270] Start()
1023 20:15:34.605533 (+ 16us) write_op.cc:276] Timestamp: P: 1761250525933629 usec, L: 0
1023 20:15:34.605648 (+ 115us) write_op.cc:270] Start()
1023 20:15:34.605677 (+ 29us) write_op.cc:276] Timestamp: P: 1761250525934368 usec, L: 0
1023 20:15:34.605798 (+ 121us) write_op.cc:270] Start()
1023 20:15:34.605829 (+ 31us) write_op.cc:276] Timestamp: P: 1761250525934894 usec, L: 0
1023 20:15:34.605947 (+ 118us) write_op.cc:270] Start()
1023 20:15:34.605970 (+ 23us) write_op.cc:276] Timestamp: P: 1761250525958575 usec, L: 0
1023 20:15:34.606103 (+ 133us) write_op.cc:270] Start()
1023 20:15:34.606119 (+ 16us) write_op.cc:276] Timestamp: P: 1761250526167074 usec, L: 0
1023 20:15:34.606247 (+ 128us) write_op.cc:270] Start()
1023 20:15:34.606264 (+ 17us) write_op.cc:276] Timestamp: P: 1761250526324972 usec, L: 0
1023 20:15:34.606384 (+ 120us) write_op.cc:270] Start()
1023 20:15:34.606401 (+ 17us) write_op.cc:276] Timestamp: P: 1761250526361686 usec, L: 0
1023 20:15:34.606560 (+ 159us) write_op.cc:270] Start()
1023 20:15:34.606580 (+ 20us) write_op.cc:276] Timestamp: P: 1761250526453287 usec, L: 0
1023 20:15:34.606838 (+ 258us) write_op.cc:270] Start()
1023 20:15:34.606860 (+ 22us) write_op.cc:276] Timestamp: P: 1761250526458644 usec, L: 0
1023 20:15:34.609361 (+ 2501us) write_op.cc:270] Start()
1023 20:15:34.612571 (+ 3210us) write_op.cc:276] Timestamp: P: 1761250526463143 usec, L: 0
1023 20:15:34.612781 (+ 210us) write_op.cc:270] Start()
1023 20:15:34.612804 (+ 23us) write_op.cc:276] Timestamp: P: 1761250526463752 usec, L: 0
1023 20:15:34.612958 (+ 154us) write_op.cc:270] Start()
1023 20:15:34.612977 (+ 19us) write_op.cc:276] Timestamp: P: 1761250526518781 usec, L: 0
1023 20:15:34.613127 (+ 150us) write_op.cc:270] Start()
1023 20:15:34.613147 (+ 20us) write_op.cc:276] Timestamp: P: 1761250526542966 usec, L: 0
1023 20:15:34.613291 (+ 144us) write_op.cc:270] Start()
1023 20:15:34.613310 (+ 19us) write_op.cc:276] Timestamp: P: 1761250526554851 usec, L: 0
1023 20:15:34.613428 (+ 118us) write_op.cc:270] Start()
1023 20:15:34.613446 (+ 18us) write_op.cc:276] Timestamp: P: 1761250526559597 usec, L: 0
1023 20:15:34.613564 (+ 118us) write_op.cc:270] Start()
1023 20:15:34.613582 (+ 18us) write_op.cc:276] Timestamp: P: 1761250526571439 usec, L: 0
1023 20:15:34.613707 (+ 125us) write_op.cc:270] Start()
1023 20:15:34.613725 (+ 18us) write_op.cc:276] Timestamp: P: 1761250526996031 usec, L: 0
1023 20:15:34.613866 (+ 141us) write_op.cc:270] Start()
1023 20:15:34.613906 (+ 40us) write_op.cc:276] Timestamp: P: 1761250527014981 usec, L: 0
1023 20:15:34.614046 (+ 140us) write_op.cc:270] Start()
1023 20:15:34.614065 (+ 19us) write_op.cc:276] Timestamp: P: 1761250527607566 usec, L: 0
1023 20:15:34.614178 (+ 113us) write_op.cc:270] Start()
1023 20:15:34.614196 (+ 18us) write_op.cc:276] Timestamp: P: 1761250528119190 usec, L: 0
1023 20:15:34.614311 (+ 115us) write_op.cc:270] Start()
1023 20:15:34.614328 (+ 17us) write_op.cc:276] Timestamp: P: 1761250528315099 usec, L: 0
1023 20:15:34.618495 (+ 4167us) write_op.cc:270] Start()
1023 20:15:34.618524 (+ 29us) write_op.cc:276] Timestamp: P: 1761250528327659 usec, L: 0
1023 20:15:34.618833 (+ 309us) write_op.cc:270] Start()
1023 20:15:34.618861 (+ 28us) write_op.cc:276] Timestamp: P: 1761250528351355 usec, L: 0
1023 20:15:34.619027 (+ 166us) write_op.cc:270] Start()
1023 20:15:34.619049 (+ 22us) write_op.cc:276] Timestamp: P: 1761250528395861 usec, L: 0
1023 20:15:34.619209 (+ 160us) write_op.cc:270] Start()
1023 20:15:34.619232 (+ 23us) write_op.cc:276] Timestamp: P: 1761250528407700 usec, L: 0
1023 20:15:34.619367 (+ 135us) write_op.cc:270] Start()
1023 20:15:34.619385 (+ 18us) write_op.cc:276] Timestamp: P: 1761250528456335 usec, L: 0
1023 20:15:34.619516 (+ 131us) write_op.cc:270] Start()
1023 20:15:34.619535 (+ 19us) write_op.cc:276] Timestamp: P: 1761250528653129 usec, L: 0
1023 20:15:34.619658 (+ 123us) write_op.cc:270] Start()
1023 20:15:34.619682 (+ 24us) write_op.cc:276] Timestamp: P: 1761250528691556 usec, L: 0
1023 20:15:34.619797 (+ 115us) write_op.cc:270] Start()
1023 20:15:34.619822 (+ 25us) write_op.cc:276] Timestamp: P: 1761250528793933 usec, L: 0
1023 20:15:34.619942 (+ 120us) write_op.cc:270] Start()
1023 20:15:34.619961 (+ 19us) write_op.cc:276] Timestamp: P: 1761250528896671 usec, L: 0
1023 20:15:34.620093 (+ 132us) write_op.cc:270] Start()
1023 20:15:34.620113 (+ 20us) write_op.cc:276] Timestamp: P: 1761250528898189 usec, L: 0
1023 20:15:34.620240 (+ 127us) write_op.cc:270] Start()
1023 20:15:34.620258 (+ 18us) write_op.cc:276] Timestamp: P: 1761250528899201 usec, L: 0
1023 20:15:34.668624 (+ 48366us) write_op.cc:270] Start()
1023 20:15:34.668676 (+ 52us) write_op.cc:276] Timestamp: P: 1761250529097663 usec, L: 0
1023 20:15:34.668941 (+ 265us) write_op.cc:270] Start()
1023 20:15:34.668964 (+ 23us) write_op.cc:276] Timestamp: P: 1761250529106437 usec, L: 0
1023 20:15:34.669133 (+ 169us) write_op.cc:270] Start()
1023 20:15:34.669153 (+ 20us) write_op.cc:276] Timestamp: P: 1761250529139555 usec, L: 0
1023 20:15:34.669304 (+ 151us) write_op.cc:270] Start()
1023 20:15:34.669321 (+ 17us) write_op.cc:276] Timestamp: P: 1761250529173398 usec, L: 0
1023 20:15:34.669448 (+ 127us) write_op.cc:270] Start()
1023 20:15:34.669493 (+ 45us) write_op.cc:276] Timestamp: P: 1761250529191265 usec, L: 0
1023 20:15:34.669626 (+ 133us) write_op.cc:270] Start()
1023 20:15:34.669641 (+ 15us) write_op.cc:276] Timestamp: P: 1761250529199785 usec, L: 0
1023 20:15:34.669862 (+ 221us) write_op.cc:270] Start()
1023 20:15:34.669888 (+ 26us) write_op.cc:276] Timestamp: P: 1761250529515769 usec, L: 0
1023 20:15:34.723675 (+ 53787us) write_op.cc:270] Start()
1023 20:15:34.723728 (+ 53us) write_op.cc:276] Timestamp: P: 1761250530082142 usec, L: 0
1023 20:15:34.747892 (+ 24164us) log.cc:844] Serialized 52643 byte log entry
1023 20:15:34.750078 (+ 2186us) raft_consensus.cc:1645] Marking committed up to 740
1023 20:15:34.750362 (+ 284us) log.cc:844] Serialized 15 byte log entry
1023 20:15:34.809474 (+ 59112us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.809485 (+ 11us) write_op.cc:454] Released schema lock
1023 20:15:34.813529 (+ 4044us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.813535 (+ 6us) write_op.cc:454] Released schema lock
1023 20:15:34.839570 (+ 26035us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.839578 (+ 8us) write_op.cc:454] Released schema lock
1023 20:15:34.846718 (+ 7140us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.846725 (+ 7us) write_op.cc:454] Released schema lock
1023 20:15:34.848394 (+ 1669us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.848399 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.849526 (+ 1127us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.849530 (+ 4us) write_op.cc:454] Released schema lock
1023 20:15:34.850115 (+ 585us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.850120 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.873169 (+ 23049us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.873180 (+ 11us) write_op.cc:454] Released schema lock
1023 20:15:34.874905 (+ 1725us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.874911 (+ 6us) write_op.cc:454] Released schema lock
1023 20:15:34.876578 (+ 1667us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.876585 (+ 7us) write_op.cc:454] Released schema lock
1023 20:15:34.877345 (+ 760us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.877391 (+ 46us) write_op.cc:454] Released schema lock
1023 20:15:34.878056 (+ 665us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.878061 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.883454 (+ 5393us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.883459 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.885301 (+ 1842us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.885307 (+ 6us) write_op.cc:454] Released schema lock
1023 20:15:34.893158 (+ 7851us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.893166 (+ 8us) write_op.cc:454] Released schema lock
1023 20:15:34.895153 (+ 1987us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.895158 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.895948 (+ 790us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.895953 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.897979 (+ 2026us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.897984 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.898647 (+ 663us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.898651 (+ 4us) write_op.cc:454] Released schema lock
1023 20:15:34.910660 (+ 12009us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.910669 (+ 9us) write_op.cc:454] Released schema lock
1023 20:15:34.914290 (+ 3621us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.914295 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.919100 (+ 4805us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.919106 (+ 6us) write_op.cc:454] Released schema lock
1023 20:15:34.922472 (+ 3366us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.922476 (+ 4us) write_op.cc:454] Released schema lock
1023 20:15:34.925147 (+ 2671us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.925152 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.930986 (+ 5834us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.930991 (+ 5us) write_op.cc:454] Released schema lock
1023 20:15:34.937422 (+ 6431us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.937428 (+ 6us) write_op.cc:454] Released schema lock
1023 20:15:35.096538 (+159110us) raft_consensus.cc:1697] Filling consensus response to leader.
1023 20:15:35.096595 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1023 20:15:35.096623 (+ 28us) raft_consensus.cc:1685] finished
1023 20:15:35.096628 (+ 5us) raft_consensus.cc:1691] UpdateReplicas() finished
1023 20:15:35.096777 (+ 149us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:34.389753 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.390007 (+ 254us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.390016 (+ 9us) write_op.cc:435] Acquired schema lock
1023 20:15:34.390019 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:34.390278 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.390297 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.390303 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:34.390415 (+ 112us) tablet.cc:699] Row locks acquired
1023 20:15:34.390418 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:34.785287 (+394869us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.793377 (+ 8090us) write_op.cc:301] APPLY: starting
1023 20:15:34.793400 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.793705 (+ 305us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.793709 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.793825 (+ 116us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.794001 (+ 176us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:34.794129 (+ 128us) write_op.cc:312] APPLY: finished
1023 20:15:34.794420 (+ 291us) log.cc:844] Serialized 23 byte log entry
1023 20:15:34.800753 (+ 6333us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.800775 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:34.800784 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:34.800870 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.800874 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:34.422754 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.422889 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.422896 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:34.422898 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:34.423008 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.423047 (+ 39us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.423053 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:34.423122 (+ 69us) tablet.cc:699] Row locks acquired
1023 20:15:34.423124 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:34.793730 (+370606us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.794307 (+ 577us) write_op.cc:301] APPLY: starting
1023 20:15:34.794325 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.794369 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.794373 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.794441 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.794470 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:34.794531 (+ 61us) write_op.cc:312] APPLY: finished
1023 20:15:34.794730 (+ 199us) log.cc:844] Serialized 23 byte log entry
1023 20:15:34.794886 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.794911 (+ 25us) write_op.cc:454] Released schema lock
1023 20:15:34.794924 (+ 13us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1023 20:15:34.425657 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.425766 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.425772 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:34.425775 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:34.425979 (+ 204us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.425996 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.426001 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:34.426153 (+ 152us) tablet.cc:699] Row locks acquired
1023 20:15:34.426155 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:34.809614 (+383459us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.809718 (+ 104us) write_op.cc:301] APPLY: starting
1023 20:15:34.809735 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.809800 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.809802 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.809880 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.809925 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:34.809955 (+ 30us) write_op.cc:312] APPLY: finished
1023 20:15:34.810046 (+ 91us) log.cc:844] Serialized 29 byte log entry
1023 20:15:34.810406 (+ 360us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.810429 (+ 23us) write_op.cc:454] Released schema lock
1023 20:15:34.810437 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:34.810509 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.810513 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:34.426938 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.427037 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.427043 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:34.427045 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:34.427141 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.427153 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.427158 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:34.427248 (+ 90us) tablet.cc:699] Row locks acquired
1023 20:15:34.427250 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:34.809701 (+382451us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.810599 (+ 898us) write_op.cc:301] APPLY: starting
1023 20:15:34.810612 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.810648 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.810650 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.810707 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.810736 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:34.810770 (+ 34us) write_op.cc:312] APPLY: finished
1023 20:15:34.810836 (+ 66us) log.cc:844] Serialized 29 byte log entry
1023 20:15:34.811112 (+ 276us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.811126 (+ 14us) write_op.cc:454] Released schema lock
1023 20:15:34.811132 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:34.811182 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.811185 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:34.427601 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.427691 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.427697 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:34.427699 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:34.427769 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.427780 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.427784 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:34.427854 (+ 70us) tablet.cc:699] Row locks acquired
1023 20:15:34.427856 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:34.809745 (+381889us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.811304 (+ 1559us) write_op.cc:301] APPLY: starting
1023 20:15:34.811314 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.811344 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.811347 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.811450 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.811476 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:34.811507 (+ 31us) write_op.cc:312] APPLY: finished
1023 20:15:34.811617 (+ 110us) log.cc:844] Serialized 29 byte log entry
1023 20:15:34.811675 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.811688 (+ 13us) write_op.cc:454] Released schema lock
1023 20:15:34.811693 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
1023 20:15:34.428156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.428238 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.428243 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:34.428245 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:34.428326 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.428337 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.428342 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:34.428426 (+ 84us) tablet.cc:699] Row locks acquired
1023 20:15:34.428429 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:34.813614 (+385185us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.813992 (+ 378us) write_op.cc:301] APPLY: starting
1023 20:15:34.814005 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.814043 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.814046 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.814315 (+ 269us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.814343 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:34.814397 (+ 54us) write_op.cc:312] APPLY: finished
1023 20:15:34.814481 (+ 84us) log.cc:844] Serialized 29 byte log entry
1023 20:15:34.814565 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.814578 (+ 13us) write_op.cc:454] Released schema lock
1023 20:15:34.814584 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:34.814631 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.814634 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:34.428771 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.428867 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.428871 (+ 4us) write_op.cc:435] Acquired schema lock
1023 20:15:34.428873 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:34.428944 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.428953 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.428957 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:34.429010 (+ 53us) tablet.cc:699] Row locks acquired
1023 20:15:34.429012 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:34.813681 (+384669us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.814696 (+ 1015us) write_op.cc:301] APPLY: starting
1023 20:15:34.814706 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.814735 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.814737 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.814852 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.814877 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups
I20251023 20:15:35.287099 23786 tablet_bootstrap.cc:492] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.287300 23786 ts_tablet_manager.cc:1403] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.001s
I20251023 20:15:35.288362 23786 raft_consensus.cc:359] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.288595 23786 raft_consensus.cc:385] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.288700 23786 raft_consensus.cc:740] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.288962 23786 consensus_queue.cc:260] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.289641 23786 ts_tablet_manager.cc:1434] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.290004 23786 tablet_bootstrap.cc:492] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.291163 23216 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3] alter complete (version 2)
I20251023 20:15:35.291486 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13] alter complete (version 1)
I20251023 20:15:35.291618 23790 consensus_queue.cc:260] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.292260 23790 ts_tablet_manager.cc:1434] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251023 20:15:35.292557 23790 tablet_bootstrap.cc:492] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.294160 23790 tablet_bootstrap.cc:654] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.295349 23786 tablet_bootstrap.cc:654] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.296217 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.296430 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.296603 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 26a680ad3aed487593bd1b63004dbbfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:30 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.296756 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.296893 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.297755 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9047d641796b4f20b0de82b277ce877c) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:35.298235 23219 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 94631fca1b834bc89791cb07fdc93523 consensus state has no leader
I20251023 20:15:35.299082 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.299760 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet ea3149f906064f8c8167292ec86540fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:35.300215 23211 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 26a680ad3aed487593bd1b63004dbbfc consensus state has no leader
I20251023 20:15:35.300956 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.300985 23786 tablet_bootstrap.cc:492] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.301136 23786 ts_tablet_manager.cc:1403] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251023 20:15:35.301453 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.301724 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.301909 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.302112 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet a86a25659ec44d7c825783036342ed45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.302284 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet a8eaa2167f82475ca2f6a110aba288ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.302367 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ed0aa7e36e64843ad6b2ad4b415832a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.302379 23787 tablet_bootstrap.cc:492] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.302299 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.302500 23787 ts_tablet_manager.cc:1403] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.003s
I20251023 20:15:35.302300 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet fb01b3f4ff7f41e98063ebf76abeace6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.302525 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 48436f3d8f1443db990a2d10b668b70a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.302641 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 352812b56f8d4feb818a364d6adfd86d with delete_type TABLET_DATA_DELETED (replaced by tablet 19da816a483a4d5b9dc768ef697d54ec) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.303089 24011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 3 candidate_status { last_received { term: 2 index: 741 } } ignore_live_leader: false dest_uuid: "6545140d94b7485aa81c51ad233355cd" is_pre_election: true
I20251023 20:15:35.303277 23787 raft_consensus.cc:359] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.303943 23787 raft_consensus.cc:385] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.304003 23787 raft_consensus.cc:740] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.304217 23787 consensus_queue.cc:260] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.304916 23787 ts_tablet_manager.cc:1434] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:35.305357 23787 tablet_bootstrap.cc:492] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.307329 25511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "690ffde75cd745b6a87925967d54471a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:35.307544 25511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a [sys.catalog]: This master's current role is: LEADER
W20251023 20:15:35.315822 23219 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 196) took 2277 ms (client timeout 1000 ms). Trace:
I20251023 20:15:35.316224 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 26a680ad3aed487593bd1b63004dbbfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:30 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.316438 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a54470d088748b0be22ab9db5a25f87 with delete_type TABLET_DATA_DELETED (replaced by tablet 3650822dd30c4c8e97982636ac692eae) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.316601 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet a636a4cdeb404e9f806d8e166ea57938 with delete_type TABLET_DATA_DELETED (replaced by tablet eb5a6d9a4509431f97b69a3b138b5829) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.316751 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet af899c01b4af46d39d27f984cc3d5940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.316859 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet f103bece2fa240c19c23ac51046c13ed with delete_type TABLET_DATA_DELETED (replaced by tablet a841ffdc6cdd432dac7afe2e2b53f224) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.316887 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 600f0110f8bd4b05983d5922d871c266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.317008 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 6480cdc1afda4671a346044f232b083e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.317394 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 514cb3b1ed024041ac160a7d9acff618 with delete_type TABLET_DATA_DELETED (replaced by tablet 370a790a053446b2b06314c7db242270) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.317623 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 72408899f710470baaf6fdb268cb04c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.317785 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bfbf0acdc84f2f8fe553694ee9897c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.317927 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.318068 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 05933cb9433d48d285b2db82b289bdc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:35.327853 23786 raft_consensus.cc:359] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.328064 23786 raft_consensus.cc:385] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.328120 23786 raft_consensus.cc:740] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.328307 23786 consensus_queue.cc:260] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.328469 24908 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d220aba7c9c0400398ffde917376ee48" instance_seqno: 1761250513291724) as {username='slave'} at 127.22.127.129:48475; Asking this server to re-register.
I20251023 20:15:35.328836 23786 ts_tablet_manager.cc:1434] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.028s user 0.000s sys 0.000s
I20251023 20:15:35.329160 23786 tablet_bootstrap.cc:492] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.330749 23786 tablet_bootstrap.cc:654] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.331791 23787 tablet_bootstrap.cc:654] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.332435 23790 tablet_bootstrap.cc:492] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.332633 23790 ts_tablet_manager.cc:1403] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20251023 20:15:35.332899 25464 raft_consensus.cc:493] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.333024 25464 raft_consensus.cc:515] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.333638 25464 leader_election.cc:290] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.333657 23790 raft_consensus.cc:359] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.333926 23790 raft_consensus.cc:385] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.333976 23446 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:35.334005 23790 raft_consensus.cc:740] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.334270 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a744214a24cc4bf28654ffa417ce0d3d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.334295 23790 consensus_queue.cc:260] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.334463 23670 raft_consensus.cc:2468] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.334905 23470 leader_election.cc:304] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.334975 23790 ts_tablet_manager.cc:1434] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.335302 25464 raft_consensus.cc:2804] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.335372 25464 raft_consensus.cc:493] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.335427 25464 raft_consensus.cc:3060] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.335785 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a744214a24cc4bf28654ffa417ce0d3d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.335955 23393 raft_consensus.cc:2468] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.336088 23790 tablet_bootstrap.cc:492] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.338018 23790 tablet_bootstrap.cc:654] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.338526 25492 tablet_replica.cc:333] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:35.338701 25492 raft_consensus.cc:2243] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.338843 25492 raft_consensus.cc:2272] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.339386 25492 ts_tablet_manager.cc:1916] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:35.341578 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 00ec606cbf834ccd8762e034b520e58e (table default.Table_c709570632244bd3af291b4e143d2eb5 [id=7dd141d21059418baa0ffa21252de2e7]) successfully deleted
I20251023 20:15:35.346087 23227 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 3 candidate_status { last_received { term: 2 index: 741 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a" is_pre_election: true
I20251023 20:15:35.350170 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 40c659c273c449939d0dd708d39b393f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.350431 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.350616 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.350776 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.350867 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet 8317a54a5f7b4afe8da9d336268efe43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.350937 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.351024 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.351310 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.351468 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a4f012469fa4e8e827e485287ad76dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.351747 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.352033 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 7f2a4f1d52984af58d6286671389b6a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9047d641796b4f20b0de82b277ce877c) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.352348 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet e194d03a6aa946fbb2c79d91b196440b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.352738 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.353165 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e36bb5e61d34b35a373bd6a51a7ac5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.353466 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 9a6a2d2b25824dbdafaf2a9cabb0f061 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.353653 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.354009 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet ea3149f906064f8c8167292ec86540fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:31 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.354199 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 453eea5cdd2e47f29577c61b40ee59c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.354450 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 76380b341a304d6ba31648fbd1548f05 with delete_type TABLET_DATA_DELETED (replaced by tablet e289b51f48a942fe91b95f528130f89b) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.354733 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet ca11ddd5d97544cf916b87fd3dff219d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.355001 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f422575c7c4251879130501ab721fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.355286 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet cbacaf2fea24405caaf84d59b625fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.355598 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 83dae34129794f49b43038d3b1636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.355901 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e5720c96730425dafb550094cf936cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
W20251023 20:15:35.356277 23219 rpcz_store.cc:269] 1023 20:15:33.038050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.263475 (+225425us) service_pool.cc:225] Handling call
1023 20:15:35.315778 (+2052303us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.157916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.159525 (+ 1609us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.159532 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:35.159535 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:35.159663 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.159678 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.159683 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:35.159749 (+ 66us) tablet.cc:699] Row locks acquired
1023 20:15:35.159751 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:35.159796 (+ 45us) write_op.cc:270] Start()
1023 20:15:35.159836 (+ 40us) write_op.cc:276] Timestamp: P: 1761250535159745 usec, L: 0
1023 20:15:35.159840 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.160055 (+ 215us) log.cc:844] Serialized 356 byte log entry
1023 20:15:35.185985 (+ 25930us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.186622 (+ 637us) write_op.cc:301] APPLY: starting
1023 20:15:35.186641 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.186692 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.186740 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.186798 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.186831 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:35.186838 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.186931 (+ 93us) log.cc:844] Serialized 23 byte log entry
1023 20:15:35.186993 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.187015 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:35.187022 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:35.187092 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.187096 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:35.285700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.285830 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.285837 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:35.285840 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:35.285946 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.285965 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.285970 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:35.286037 (+ 67us) tablet.cc:699] Row locks acquired
1023 20:15:35.286039 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:35.286080 (+ 41us) write_op.cc:270] Start()
1023 20:15:35.286117 (+ 37us) write_op.cc:276] Timestamp: P: 1761250535286049 usec, L: 0
1023 20:15:35.286121 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.286321 (+ 200us) log.cc:844] Serialized 356 byte log entry
1023 20:15:35.290723 (+ 4402us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.290857 (+ 134us) write_op.cc:301] APPLY: starting
1023 20:15:35.290873 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.290918 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.290951 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.291012 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.291045 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:35.291052 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.291137 (+ 85us) log.cc:844] Serialized 23 byte log entry
1023 20:15:35.291214 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.291236 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:35.291243 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:35.291318 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.291322 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":155361,"reported_tablets":200,"spinlock_wait_cycles":768,"child_traces":[["op",{"apply.queue_time_us":527,"apply.run_cpu_time_us":470,"apply.run_wall_time_us":470,"num_ops":1,"prepare.queue_time_us":12105,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":2657,"raft.queue_time_us":6315,"raft.run_cpu_time_us":522,"raft.run_wall_time_us":521,"replication_time_us":26110}],["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":451,"num_ops":1,"prepare.queue_time_us":5550,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":736,"raft.queue_time_us":429,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":288,"replication_time_us":4568}]]}
I20251023 20:15:35.358188 24780 ts_tablet_manager.cc:1929] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:35.358314 24780 log.cc:1199] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/2532c8567db14901b383f42e5f04b4c2
I20251023 20:15:35.358664 24780 ts_tablet_manager.cc:1950] T 2532c8567db14901b383f42e5f04b4c2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:35.358901 23392 raft_consensus.cc:2468] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:35.360845 24902 ts_manager.cc:194] Registered new tserver with Master: d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.360826 23532 raft_consensus.cc:2468] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:35.361382 23665 raft_consensus.cc:2468] T 92576756ec574ceaa4c81f435cb72301 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:35.364604 23333 leader_election.cc:304] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:35.364949 25172 raft_consensus.cc:2804] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.365060 25172 raft_consensus.cc:697] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:35.365314 25172 consensus_queue.cc:237] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.365278 24831 leader_election.cc:304] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae5a310b4a514c0f812bda133c756062; no voters: 6545140d94b7485aa81c51ad233355cd, 690ffde75cd745b6a87925967d54471a
I20251023 20:15:35.365685 23471 leader_election.cc:304] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.366386 23333 leader_election.cc:304] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.366655 25462 raft_consensus.cc:2804] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.366739 25235 raft_consensus.cc:2804] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.366766 25462 raft_consensus.cc:697] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:35.366837 25235 raft_consensus.cc:697] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:35.366987 25462 consensus_queue.cc:237] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.367058 25235 consensus_queue.cc:237] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.369235 25282 ts_tablet_manager.cc:1929] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:35.369372 25282 log.cc:1199] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/857da3642f0846ce92aaed0491e89644
I20251023 20:15:35.369696 25282 ts_tablet_manager.cc:1950] T 857da3642f0846ce92aaed0491e89644 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:35.377223 25488 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 20:15:35.379246 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 46f700ea3db04a4bad539c943e850a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.379469 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.379604 25465 raft_consensus.cc:493] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.379714 25465 raft_consensus.cc:515] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.380311 25465 leader_election.cc:290] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.381018 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b593b020b34a969f77d2f68178e89d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:35.377071 23219 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 171) took 1169 ms (client timeout 1000 ms). Trace:
I20251023 20:15:35.381278 23665 raft_consensus.cc:2468] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
W20251023 20:15:35.381332 23219 rpcz_store.cc:269] 1023 20:15:34.207444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.376983 (+1169539us) service_pool.cc:212] Skipping call since client already timed out
1023 20:15:35.377028 (+ 45us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20251023 20:15:35.381671 23470 leader_election.cc:304] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.381953 25465 raft_consensus.cc:2804] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.382027 25465 raft_consensus.cc:493] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.382094 25465 raft_consensus.cc:3060] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.383111 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b593b020b34a969f77d2f68178e89d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.383307 23392 raft_consensus.cc:2468] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.383787 24297 consensus_queue.cc:1048] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.398183 23516 tablet_service.cc:1552] Processing DeleteTablet for tablet d62dbf044acd4459b26d2e7e50f6206e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.398351 23511 tablet_service.cc:1552] Processing DeleteTablet for tablet 01fa5f9c700b458ea147373d78551573 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.399775 23504 tablet_service.cc:1552] Processing DeleteTablet for tablet 377de4ea0bd6489284fac6b0fab52c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:26 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.399935 23787 tablet_bootstrap.cc:492] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.400025 23497 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.400206 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.400553 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 5acdc66df9c748a1a6300962f62fcf36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.400903 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.401140 23787 ts_tablet_manager.cc:1403] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.011s
I20251023 20:15:35.401189 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.401352 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.402957 25470 consensus_queue.cc:1048] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.404187 23786 tablet_bootstrap.cc:492] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.404320 23786 ts_tablet_manager.cc:1403] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.075s user 0.017s sys 0.004s
I20251023 20:15:35.404210 23787 raft_consensus.cc:359] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.404507 23787 raft_consensus.cc:385] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.404623 23787 raft_consensus.cc:740] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.404906 23787 consensus_queue.cc:260] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.409991 25359 raft_consensus.cc:493] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.410137 25359 raft_consensus.cc:515] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.410737 25359 leader_election.cc:290] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.411389 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2c022d78cac4959984d608d64df851b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.411995 23605 leader_election.cc:343] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.412115 23786 raft_consensus.cc:359] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.412302 23786 raft_consensus.cc:385] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.412343 23786 raft_consensus.cc:740] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.412436 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2c022d78cac4959984d608d64df851b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.412541 23786 consensus_queue.cc:260] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:35.412916 23604 leader_election.cc:343] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.413028 23786 ts_tablet_manager.cc:1434] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251023 20:15:35.413030 23604 leader_election.cc:304] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.413318 23786 tablet_bootstrap.cc:492] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.413431 25359 raft_consensus.cc:2749] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.415035 23786 tablet_bootstrap.cc:654] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.417253 23790 tablet_bootstrap.cc:492] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.417606 24821 raft_consensus.cc:493] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.417726 24821 raft_consensus.cc:515] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.418215 24821 leader_election.cc:290] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:35.419013 23333 leader_election.cc:343] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 40c659c273c449939d0dd708d39b393f
I20251023 20:15:35.419509 23790 ts_tablet_manager.cc:1403] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.015s
I20251023 20:15:35.420547 23790 raft_consensus.cc:359] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.420671 23787 ts_tablet_manager.cc:1434] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251023 20:15:35.421501 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40c659c273c449939d0dd708d39b393f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.421917 23530 raft_consensus.cc:2468] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.422035 23787 tablet_bootstrap.cc:492] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.420711 23790 raft_consensus.cc:385] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.422407 23790 raft_consensus.cc:740] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.423161 23790 consensus_queue.cc:260] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.423324 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.423772 23514 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.424105 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.424316 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.424649 23790 ts_tablet_manager.cc:1434] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:35.424676 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:35.425166 23333 leader_election.cc:304] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.426465 23790 tablet_bootstrap.cc:492] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.428337 25462 consensus_queue.cc:1048] T 882f7bf08e2f4b5382702c3d6eaf201e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.428889 25040 raft_consensus.cc:2804] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.428977 25040 raft_consensus.cc:493] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.429052 25040 raft_consensus.cc:3060] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.430617 23790 tablet_bootstrap.cc:654] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.430812 25235 consensus_queue.cc:1048] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 20:15:35.437745 23786 tablet_bootstrap.cc:492] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.438148 23786 ts_tablet_manager.cc:1403] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251023 20:15:35.439154 23786 raft_consensus.cc:359] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.439504 23786 raft_consensus.cc:385] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.439576 23786 raft_consensus.cc:740] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.439780 23786 consensus_queue.cc:260] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.440263 23786 ts_tablet_manager.cc:1434] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.440515 23786 tablet_bootstrap.cc:492] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.442903 23786 tablet_bootstrap.cc:654] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:35.444067 23214 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 194) took 4554 ms (client timeout 1000 ms). Trace:
I20251023 20:15:35.454730 23790 tablet_bootstrap.cc:492] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.454887 23790 ts_tablet_manager.cc:1403] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20251023 20:15:35.455608 23790 raft_consensus.cc:359] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.455785 23790 raft_consensus.cc:385] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.455832 23790 raft_consensus.cc:740] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.456071 23790 consensus_queue.cc:260] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.456528 23790 ts_tablet_manager.cc:1434] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:35.456796 23790 tablet_bootstrap.cc:492] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.457747 23786 tablet_bootstrap.cc:492] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.457983 23786 ts_tablet_manager.cc:1403] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20251023 20:15:35.458447 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 168) took 6681 ms. Trace:
I20251023 20:15:35.458539 24780 rpcz_store.cc:276] 1023 20:15:28.777377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.777537 (+ 160us) service_pool.cc:225] Handling call
1023 20:15:35.253352 (+6475815us) ts_tablet_manager.cc:1151] Deleting tablet 2532c8567db14901b383f42e5f04b4c2
1023 20:15:35.253359 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.257116 (+ 3757us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.358061 (+100945us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.458420 (+100359us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6475659}
I20251023 20:15:35.458711 23786 raft_consensus.cc:359] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.458876 23786 raft_consensus.cc:385] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.458927 23786 raft_consensus.cc:740] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.459144 23786 consensus_queue.cc:260] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.459609 23786 ts_tablet_manager.cc:1434] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:35.458900 24780 tablet_replica.cc:333] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:35.459755 23787 tablet_bootstrap.cc:654] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.459852 24780 raft_consensus.cc:2243] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.459901 23790 tablet_bootstrap.cc:654] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.459985 24780 raft_consensus.cc:2272] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.460493 24780 ts_tablet_manager.cc:1916] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:35.461269 25262 consensus_meta.cc:287] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.750s user 0.000s sys 0.003s
I20251023 20:15:35.461361 25262 raft_consensus.cc:515] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.461916 25262 leader_election.cc:290] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.467151 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76380b341a304d6ba31648fbd1548f05" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:35.467376 23666 raft_consensus.cc:3060] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.468308 23664 raft_consensus.cc:2468] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:35.469223 25040 raft_consensus.cc:515] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.469883 25040 leader_election.cc:290] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.470628 23790 tablet_bootstrap.cc:492] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:35.470813 23333 leader_election.cc:343] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 40c659c273c449939d0dd708d39b393f
I20251023 20:15:35.471771 23790 ts_tablet_manager.cc:1403] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20251023 20:15:35.472463 23790 raft_consensus.cc:359] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.472604 23790 raft_consensus.cc:385] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.472646 23790 raft_consensus.cc:740] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.473882 23790 consensus_queue.cc:260] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.474167 23787 tablet_bootstrap.cc:492] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.474303 23787 ts_tablet_manager.cc:1403] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
I20251023 20:15:35.474359 23790 ts_tablet_manager.cc:1434] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251023 20:15:35.474599 23790 tablet_bootstrap.cc:492] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.474592 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 365) took 1948 ms. Trace:
I20251023 20:15:35.474689 25282 rpcz_store.cc:276] 1023 20:15:33.525742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.525804 (+ 62us) service_pool.cc:225] Handling call
1023 20:15:35.152444 (+1626640us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:35.152455 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.259615 (+107160us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.364553 (+104938us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.474552 (+109999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621039}
I20251023 20:15:35.475117 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 366) took 1949 ms. Trace:
I20251023 20:15:35.475065 23787 raft_consensus.cc:359] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.475232 23787 raft_consensus.cc:385] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.475301 23787 raft_consensus.cc:740] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.475196 25282 rpcz_store.cc:276] 1023 20:15:33.525885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.525961 (+ 76us) service_pool.cc:225] Handling call
1023 20:15:35.474945 (+1948984us) ts_tablet_manager.cc:1151] Deleting tablet a841ffdc6cdd432dac7afe2e2b53f224
1023 20:15:35.474955 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.475080 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1943498}
I20251023 20:15:35.475492 25282 tablet_replica.cc:333] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:35.476014 23787 consensus_queue.cc:260] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.476563 23787 ts_tablet_manager.cc:1434] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:35.476766 23790 tablet_bootstrap.cc:654] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.476950 23787 tablet_bootstrap.cc:492] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.479027 23787 tablet_bootstrap.cc:654] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.480170 25262 raft_consensus.cc:493] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.480326 25262 raft_consensus.cc:515] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.481827 25262 leader_election.cc:290] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.482282 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53f33020319d48c791e1156c67b55be6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.482508 23664 raft_consensus.cc:2468] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.482998 23470 leader_election.cc:304] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.483086 23790 tablet_bootstrap.cc:492] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.483206 23790 ts_tablet_manager.cc:1403] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20251023 20:15:35.483418 25262 raft_consensus.cc:2804] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.483493 25262 raft_consensus.cc:493] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.483559 25262 raft_consensus.cc:3060] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.484441 23790 raft_consensus.cc:359] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.485237 23790 raft_consensus.cc:385] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.485419 25470 raft_consensus.cc:493] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.485320 23790 raft_consensus.cc:740] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.485551 25470 raft_consensus.cc:515] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.485649 23790 consensus_queue.cc:260] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.486109 23790 ts_tablet_manager.cc:1434] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:35.486178 25470 leader_election.cc:290] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.487072 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "473324b22da14618a84aa1136d081993" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.487237 23790 tablet_bootstrap.cc:492] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.487251 23664 raft_consensus.cc:2468] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.487732 23470 leader_election.cc:304] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.488013 25470 raft_consensus.cc:2804] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.488082 25470 raft_consensus.cc:493] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.488138 25470 raft_consensus.cc:3060] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.489312 23787 tablet_bootstrap.cc:492] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.489487 23787 ts_tablet_manager.cc:1403] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20251023 20:15:35.490221 23787 raft_consensus.cc:359] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.490430 23787 raft_consensus.cc:385] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.490522 23787 raft_consensus.cc:740] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.490727 23790 tablet_bootstrap.cc:654] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.476022 25282 raft_consensus.cc:2243] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.490967 25282 raft_consensus.cc:2272] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.490855 23787 consensus_queue.cc:260] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.491479 25282 ts_tablet_manager.cc:1916] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:35.491511 23787 ts_tablet_manager.cc:1434] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:35.491853 23787 tablet_bootstrap.cc:492] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.496042 23787 tablet_bootstrap.cc:654] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.499593 23790 tablet_bootstrap.cc:492] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.500324 23790 ts_tablet_manager.cc:1403] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20251023 20:15:35.502031 23790 raft_consensus.cc:359] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.502205 23790 raft_consensus.cc:385] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.502826 23790 raft_consensus.cc:740] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.503412 23790 consensus_queue.cc:260] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.504149 23790 ts_tablet_manager.cc:1434] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251023 20:15:35.504838 23790 tablet_bootstrap.cc:492] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.506933 23790 tablet_bootstrap.cc:654] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.510170 23787 tablet_bootstrap.cc:492] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.510381 23787 ts_tablet_manager.cc:1403] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20251023 20:15:35.511474 23787 raft_consensus.cc:359] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.511993 23787 raft_consensus.cc:385] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.512146 23787 raft_consensus.cc:740] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.512416 23787 consensus_queue.cc:260] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.513552 23790 tablet_bootstrap.cc:492] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.513662 23787 ts_tablet_manager.cc:1434] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251023 20:15:35.513981 23787 tablet_bootstrap.cc:492] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.514020 23790 ts_tablet_manager.cc:1403] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20251023 20:15:35.514988 23790 raft_consensus.cc:359] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.515164 23790 raft_consensus.cc:385] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.515214 23790 raft_consensus.cc:740] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.515403 23790 consensus_queue.cc:260] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.516752 23787 tablet_bootstrap.cc:654] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.517436 23790 ts_tablet_manager.cc:1434] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251023 20:15:35.517791 23790 tablet_bootstrap.cc:492] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.520289 23790 tablet_bootstrap.cc:654] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.525995 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53f33020319d48c791e1156c67b55be6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.526189 23393 raft_consensus.cc:2468] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.526448 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "473324b22da14618a84aa1136d081993" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.526633 23398 raft_consensus.cc:2468] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.530664 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76380b341a304d6ba31648fbd1548f05" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:35.530900 23392 raft_consensus.cc:3060] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:35.531718 23214 rpcz_store.cc:269] 1023 20:15:30.889978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.993570 (+103592us) service_pool.cc:225] Handling call
1023 20:15:35.444011 (+4450441us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97753,"reported_tablets":202}
I20251023 20:15:35.532321 23214 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_365837ad05624d75855461852362db1c" } modify_external_catalogs: true
I20251023 20:15:35.533242 23581 heartbeater.cc:344] Connected to a master server at 127.22.127.188:41417
I20251023 20:15:35.533414 23581 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:35.536764 23790 tablet_bootstrap.cc:492] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.536912 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40c659c273c449939d0dd708d39b393f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:35.532526 23214 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_365837ad05624d75855461852362db1c" } modify_external_catalogs: true
I20251023 20:15:35.537148 23532 raft_consensus.cc:3060] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:35.532603 23217 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 168) took 4536 ms (client timeout 1000 ms). Trace:
W20251023 20:15:35.537357 23217 rpcz_store.cc:269] 1023 20:15:30.882073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.885622 (+ 3549us) service_pool.cc:225] Handling call
1023 20:15:35.418345 (+4532723us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125290,"reported_tablets":221}
I20251023 20:15:35.537792 23217 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4" } modify_external_catalogs: true
I20251023 20:15:35.537968 23217 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_db33a3e2d0d442bb8b9fc3c9d69482b4" } modify_external_catalogs: true
I20251023 20:15:35.538590 23790 ts_tablet_manager.cc:1403] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
W20251023 20:15:35.447659 23210 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 170) took 2451 ms (client timeout 1000 ms). Trace:
I20251023 20:15:35.468003 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 2532c8567db14901b383f42e5f04b4c2 (table default.Table_bf99025eb142454fa0513fafcbe636aa [id=f52f1d83bd9345ffa08e6d3d282914e0]) successfully deleted
W20251023 20:15:35.538996 23205 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 71053.4us
I20251023 20:15:35.540033 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet a708cd011802458aa0e2991fd4b4a2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.540177 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet f9bd2b109bfa4804a372965e60ba6e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.540262 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e1b4748c0244b258960b1025da47562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.540390 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.540702 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 1822fdf1f5f44dd4a742aa3c2f8443cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.540815 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 3befdfc85bca42bb816e11d1dd832c8c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.540988 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet e418a62d00824cff89b37a7bace5c469 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541000 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 5964da70989b4e6cb77fe5d0a52ce9cf with delete_type TABLET_DATA_DELETED (replaced by tablet 00ec606cbf834ccd8762e034b520e58e) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541208 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet d9f73477a06542c0b215ae86716cdc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541304 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 9fc7e18054c440098dc2470143ea7fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:16 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541460 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 7645c36c983f438f8b1223256fd3ad2c with delete_type TABLET_DATA_DELETED (replaced by tablet 82ae1a94b011412c851cd96946744fc4) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541652 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ae6a6e58274b8baf952eaaefde7127 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541731 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet db758a7a7a89410f8ac1da72048fbc56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541792 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet a21ec49194e24723b31fd9c86f1f85e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541914 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e0301fb5974ec5b6e8291230a736c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bce63866d524067b87145b07347ffcf) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.541996 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 25079532547d4175802b84877790662d with delete_type TABLET_DATA_DELETED (replaced by tablet c6f3abcaa81c4646bc1432bcbd1d4fcf) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542080 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet ac55e14745534a02a2f6a422cab6b149 with delete_type TABLET_DATA_DELETED (replaced by tablet 473324b22da14618a84aa1136d081993) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542160 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet fe922c438a894a7aa04699b2d2897a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542253 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet a27f192792cf41288d69ca15ab538472 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542347 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet b90810781f724c78935a33d868ffa27d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542428 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 15ced74d0dd74e5195ed98358957d0dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:18 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542547 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 283166bbd8004ce3bca9f2ecbdbcbf48 with delete_type TABLET_DATA_DELETED (replaced by tablet 419e41d7a37f4b888cb242b089c28790) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542613 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet d951134f6773410386bc57f7f90310c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542789 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet 994a555b24e4432a84601c54d422aa5d with delete_type TABLET_DATA_DELETED (replaced by tablet c7a63d37788240f0a0c3dbfdd14edeac) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542882 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet fbab01ecb27a426188689408079eaa01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.542930 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet bb475e303d07491bbc19dc3f628bceb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:20 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.543071 23368 tablet_service.cc:1552] Processing DeleteTablet for tablet 5346a221c78342359eb6e418bd12cd24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:19 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.543078 23367 tablet_service.cc:1552] Processing DeleteTablet for tablet ea0cbd6bec674cf5831486f500cc7b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.543249 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 0438593fbbdf47cebfb64121acad104a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.543715 23369 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e8eb7603c3e46f1b15cdff7a7bbedcf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.543890 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 0858358aabff47a69dc77f4669b225c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.544028 23372 tablet_service.cc:1552] Processing DeleteTablet for tablet 66a2f3adb5d347a0adc198cd1ef93304 with delete_type TABLET_DATA_DELETED (replaced by tablet b680b1cbc103403b91a1104248ecbdcf) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:35.544478 23786 tablet_bootstrap.cc:492] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.545023 23790 raft_consensus.cc:359] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.545198 23790 raft_consensus.cc:385] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.546310 23786 tablet_bootstrap.cc:654] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.545254 23790 raft_consensus.cc:740] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.547134 23715 heartbeater.cc:344] Connected to a master server at 127.22.127.188:41417
I20251023 20:15:35.547374 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:35.547533 23790 consensus_queue.cc:260] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.548473 23790 ts_tablet_manager.cc:1434] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:35.548779 23790 tablet_bootstrap.cc:492] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.552238 23790 tablet_bootstrap.cc:654] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.493347 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 857da3642f0846ce92aaed0491e89644 (table default.Table_6311c71cac724f7f8ba71a660247f056 [id=0f5e9dcbdd7f41efab4ded041edafd7d]) successfully deleted
W20251023 20:15:35.506703 23215 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 195) took 5526 ms (client timeout 1000 ms). Trace:
W20251023 20:15:35.552780 23204 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 59458.9us
I20251023 20:15:35.552852 24297 consensus_queue.cc:1048] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:35.553066 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet a841ffdc6cdd432dac7afe2e2b53f224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a841ffdc6cdd432dac7afe2e2b53f224 already in progress: creating tablet
W20251023 20:15:35.552775 23215 rpcz_store.cc:269] 1023 20:15:29.980308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.442323 (+462015us) service_pool.cc:225] Handling call
1023 20:15:30.550173 (+107850us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a05fcfe8 00007f89971042c6 00007f8997103ff5 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:31.863608 (+1313435us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.863614 (+ 6us) write_op.cc:454] Released schema lock
1023 20:15:35.506645 (+3643031us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:31.855989 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:31.856130 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:31.856138 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:31.856140 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:31.856360 (+ 220us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:31.856374 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:31.856381 (+ 7us) tablet.cc:683] Acquiring locks for 6 operations
1023 20:15:31.856505 (+ 124us) tablet.cc:699] Row locks acquired
1023 20:15:31.856507 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:31.856551 (+ 44us) write_op.cc:270] Start()
1023 20:15:31.856592 (+ 41us) write_op.cc:276] Timestamp: P: 1761250531856514 usec, L: 0
1023 20:15:31.856595 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:31.856819 (+ 224us) log.cc:844] Serialized 2452 byte log entry
1023 20:15:31.860862 (+ 4043us) op_driver.cc:464] REPLICATION: finished
1023 20:15:31.861018 (+ 156us) write_op.cc:301] APPLY: starting
1023 20:15:31.861036 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:31.861122 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:31.861155 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:31.861266 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:31.861303 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
1023 20:15:31.861310 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:31.861405 (+ 95us) log.cc:844] Serialized 53 byte log entry
1023 20:15:31.861468 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:31.861493 (+ 25us) write_op.cc:454] Released schema lock
1023 20:15:31.861500 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":196750,"reported_tablets":241,"spinlock_wait_cycles":23977856,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":527,"apply.run_wall_time_us":535,"num_ops":6,"prepare.queue_time_us":321,"prepare.run_cpu_time_us":939,"prepare.run_wall_time_us":939,"raft.queue_time_us":448,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":304,"replication_time_us":4236}]]}
I20251023 20:15:35.554140 23215 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_a29ebb47860443d89041e72a86ac92fa" } new_table_name: "default.Table_511899f307ee49fb8612942d4b966ee2" modify_external_catalogs: true
W20251023 20:15:35.531558 23211 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.130:56879 (request call id 167) took 5772 ms (client timeout 1000 ms). Trace:
W20251023 20:15:35.555164 23211 rpcz_store.cc:269] 1023 20:15:29.759226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.406626 (+647400us) service_pool.cc:225] Handling call
1023 20:15:35.531496 (+5124870us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":111460,"reported_tablets":215}
I20251023 20:15:35.555760 23211 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_c1f42ba6bf654a09ade937991a5f85c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:35.556494 23211 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1f42ba6bf654a09ade937991a5f85c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:35.557890 23786 tablet_bootstrap.cc:492] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.558072 23786 ts_tablet_manager.cc:1403] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251023 20:15:35.558809 23786 raft_consensus.cc:359] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.558975 23786 raft_consensus.cc:385] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.559072 23786 raft_consensus.cc:740] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:35.559247 23534 consensus_meta.cc:287] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.591s user 0.001s sys 0.002s
I20251023 20:15:35.559341 23534 raft_consensus.cc:2468] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:35.559310 23786 consensus_queue.cc:260] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.560369 23786 ts_tablet_manager.cc:1434] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.560740 23786 tablet_bootstrap.cc:492] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.562867 23666 raft_consensus.cc:2468] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:35.563374 23786 tablet_bootstrap.cc:654] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.564641 23470 leader_election.cc:304] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.564908 25554 raft_consensus.cc:2804] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.565004 25554 raft_consensus.cc:697] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:35.565266 25554 consensus_queue.cc:237] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.570451 23392 raft_consensus.cc:2468] T 76380b341a304d6ba31648fbd1548f05 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:35.575584 23210 rpcz_store.cc:269] 1023 20:15:32.996229 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.218462 (+222233us) service_pool.cc:225] Handling call
1023 20:15:35.447603 (+2229141us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:34.110327 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.110447 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.110455 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:34.110457 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:34.110659 (+ 202us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.110674 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.110679 (+ 5us) tablet.cc:683] Acquiring locks for 6 operations
1023 20:15:34.110832 (+ 153us) tablet.cc:699] Row locks acquired
1023 20:15:34.110834 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:34.110870 (+ 36us) write_op.cc:270] Start()
1023 20:15:34.110904 (+ 34us) write_op.cc:276] Timestamp: P: 1761250534110822 usec, L: 0
1023 20:15:34.110908 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:34.111099 (+ 191us) log.cc:844] Serialized 2452 byte log entry
1023 20:15:34.120148 (+ 9049us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.120270 (+ 122us) write_op.cc:301] APPLY: starting
1023 20:15:34.120289 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.120353 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.120393 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.120520 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.120559 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
1023 20:15:34.120568 (+ 9us) write_op.cc:312] APPLY: finished
1023 20:15:34.120714 (+ 146us) log.cc:844] Serialized 53 byte log entry
1023 20:15:34.120790 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.120828 (+ 38us) write_op.cc:454] Released schema lock
1023 20:15:34.120837 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:34.120914 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.120918 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":85883,"reported_tablets":208,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":633,"apply.run_wall_time_us":633,"num_ops":6,"prepare.queue_time_us":200,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":882,"raft.queue_time_us":47,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":327,"replication_time_us":9204}]]}
I20251023 20:15:35.580302 23786 tablet_bootstrap.cc:492] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.581502 23333 leader_election.cc:304] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.581763 24297 raft_consensus.cc:2804] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.581892 24297 raft_consensus.cc:697] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:35.581188 23786 ts_tablet_manager.cc:1403] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20251023 20:15:35.582121 24297 consensus_queue.cc:237] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.585265 23790 tablet_bootstrap.cc:492] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.582911 23786 raft_consensus.cc:359] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.585491 23786 raft_consensus.cc:385] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.585589 23786 raft_consensus.cc:740] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.586014 25040 consensus_queue.cc:1048] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.586069 23786 consensus_queue.cc:260] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.586192 25492 ts_tablet_manager.cc:1929] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:35.586284 25492 log.cc:1199] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/514cb3b1ed024041ac160a7d9acff618
I20251023 20:15:35.586361 23790 ts_tablet_manager.cc:1403] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20251023 20:15:35.588384 23790 raft_consensus.cc:359] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.589021 23786 ts_tablet_manager.cc:1434] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:35.589354 23786 tablet_bootstrap.cc:492] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.588570 23790 raft_consensus.cc:385] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.591117 23790 raft_consensus.cc:740] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.591459 23790 consensus_queue.cc:260] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.592103 23790 ts_tablet_manager.cc:1434] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:35.592403 23790 tablet_bootstrap.cc:492] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.592933 23786 tablet_bootstrap.cc:654] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.595574 23790 tablet_bootstrap.cc:654] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.600842 23786 tablet_bootstrap.cc:492] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.601557 23786 ts_tablet_manager.cc:1403] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20251023 20:15:35.603756 25276 raft_consensus.cc:493] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.604012 25276 raft_consensus.cc:515] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.604763 25276 leader_election.cc:290] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.605867 23786 raft_consensus.cc:359] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.605957 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.606560 23605 leader_election.cc:343] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.611722 23790 tablet_bootstrap.cc:492] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.611883 23790 ts_tablet_manager.cc:1403] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20251023 20:15:35.612674 23790 raft_consensus.cc:359] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.612840 23790 raft_consensus.cc:385] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.612893 23790 raft_consensus.cc:740] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.613113 23790 consensus_queue.cc:260] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.613479 23790 ts_tablet_manager.cc:1434] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:35.613750 23790 tablet_bootstrap.cc:492] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.615418 23790 tablet_bootstrap.cc:654] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.606884 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.615974 23786 raft_consensus.cc:385] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.616051 23786 raft_consensus.cc:740] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:35.616622 23604 leader_election.cc:343] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.616791 23604 leader_election.cc:304] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.617323 25276 raft_consensus.cc:2749] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.618217 23786 consensus_queue.cc:260] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.618631 25492 ts_tablet_manager.cc:1950] T 514cb3b1ed024041ac160a7d9acff618 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:35.620930 23786 ts_tablet_manager.cc:1434] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251023 20:15:35.621313 23786 tablet_bootstrap.cc:492] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.623412 23786 tablet_bootstrap.cc:654] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.630901 23786 tablet_bootstrap.cc:492] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.631796 23790 tablet_bootstrap.cc:492] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.632489 23790 ts_tablet_manager.cc:1403] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251023 20:15:35.632735 23786 ts_tablet_manager.cc:1403] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
W20251023 20:15:35.632980 23333 consensus_peers.cc:597] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:35.638149 23786 raft_consensus.cc:359] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.639194 23790 raft_consensus.cc:359] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.640131 23790 raft_consensus.cc:385] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.640288 23786 raft_consensus.cc:385] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.640360 25276 raft_consensus.cc:493] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.640447 23790 raft_consensus.cc:740] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.640534 25276 raft_consensus.cc:515] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.640669 23790 consensus_queue.cc:260] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.640913 23786 raft_consensus.cc:740] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.641685 23786 consensus_queue.cc:260] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.641847 25276 leader_election.cc:290] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.642349 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "225ce6aa52c54f76bd56247f01219dbc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.642867 23605 leader_election.cc:343] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.643150 23790 ts_tablet_manager.cc:1434] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:35.643400 23786 ts_tablet_manager.cc:1434] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:35.643765 23790 tablet_bootstrap.cc:492] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.644193 23786 tablet_bootstrap.cc:492] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.646735 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_6ea21ba3f1684634907e258298159baf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:35.647470 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ea21ba3f1684634907e258298159baf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:35.652813 23787 tablet_bootstrap.cc:492] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.653074 23787 ts_tablet_manager.cc:1403] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.139s user 0.047s sys 0.024s
I20251023 20:15:35.654110 23787 raft_consensus.cc:359] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.654371 23787 raft_consensus.cc:385] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.654479 23787 raft_consensus.cc:740] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.654901 23787 consensus_queue.cc:260] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.652264 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 2532c8567db14901b383f42e5f04b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:17 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:35.660369 25466 consensus_meta.cc:287] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.903s user 0.004s sys 0.000s
I20251023 20:15:35.660521 25466 raft_consensus.cc:515] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.661260 25466 leader_election.cc:290] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.661409 23787 ts_tablet_manager.cc:1434] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251023 20:15:35.661780 23787 tablet_bootstrap.cc:492] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.662714 25359 raft_consensus.cc:493] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.662850 25359 raft_consensus.cc:515] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.663399 25359 leader_election.cc:290] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.664422 23787 tablet_bootstrap.cc:654] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.664718 25235 raft_consensus.cc:493] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:35.664846 25235 raft_consensus.cc:515] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:35.665555 25235 leader_election.cc:290] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.644416 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "225ce6aa52c54f76bd56247f01219dbc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.667418 23790 tablet_bootstrap.cc:654] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:35.667906 23604 leader_election.cc:343] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.668052 23604 leader_election.cc:304] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.668493 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31df5c2c40de4d88b6ab1567b84ce4ef" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.669595 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29c2c976d6a3440394f478a31abc7134" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
W20251023 20:15:35.669664 23333 leader_election.cc:343] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:35.669807 23670 raft_consensus.cc:3060] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.669992 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31df5c2c40de4d88b6ab1567b84ce4ef" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.670514 23604 leader_election.cc:343] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.670737 25359 raft_consensus.cc:2749] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:35.671020 23605 leader_election.cc:343] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.671144 23605 leader_election.cc:304] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.671727 24780 ts_tablet_manager.cc:1929] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:35.671818 24780 log.cc:1199] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/419e41d7a37f4b888cb242b089c28790
I20251023 20:15:35.672106 24780 ts_tablet_manager.cc:1950] T 419e41d7a37f4b888cb242b089c28790 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:35.672462 25359 raft_consensus.cc:2749] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.676124 23786 tablet_bootstrap.cc:654] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.685130 25466 consensus_queue.cc:1048] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.685765 25172 raft_consensus.cc:493] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.686013 25172 raft_consensus.cc:515] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.687085 25172 leader_election.cc:290] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:35.687789 23333 leader_election.cc:343] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 01fa5f9c700b458ea147373d78551573
I20251023 20:15:35.688377 25282 ts_tablet_manager.cc:1929] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:35.688519 25282 log.cc:1199] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/39a62c97e0a3459e91c24c1c2f92389a
I20251023 20:15:35.688845 25282 ts_tablet_manager.cc:1950] T 39a62c97e0a3459e91c24c1c2f92389a P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:35.691202 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29c2c976d6a3440394f478a31abc7134" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:35.691365 23392 raft_consensus.cc:3060] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.693890 23786 tablet_bootstrap.cc:492] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.694029 23786 ts_tablet_manager.cc:1403] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.000s
I20251023 20:15:35.694825 23786 raft_consensus.cc:359] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.694988 23786 raft_consensus.cc:385] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.695045 23786 raft_consensus.cc:740] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.695220 23786 consensus_queue.cc:260] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.695844 23786 ts_tablet_manager.cc:1434] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:35.696115 23786 tablet_bootstrap.cc:492] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.696723 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01fa5f9c700b458ea147373d78551573" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.696908 23530 raft_consensus.cc:2468] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:35.697758 23333 leader_election.cc:343] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:35.697887 23333 leader_election.cc:304] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.698356 23333 leader_election.cc:304] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.698380 24821 raft_consensus.cc:2749] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:35.698570 24821 raft_consensus.cc:2804] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.698632 24821 raft_consensus.cc:493] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.698685 24821 raft_consensus.cc:3060] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.702153 25359 raft_consensus.cc:493] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.702291 23786 tablet_bootstrap.cc:654] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.702311 25359 raft_consensus.cc:515] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.702904 25359 leader_election.cc:290] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.703779 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3150a8478c144d285e999d5c9270fe0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.704443 23605 leader_election.cc:343] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.704737 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3150a8478c144d285e999d5c9270fe0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:35.705278 23604 leader_election.cc:343] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.705379 23604 leader_election.cc:304] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.705749 25276 raft_consensus.cc:2749] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.707816 24297 consensus_queue.cc:1048] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.709069 25466 consensus_queue.cc:1048] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.711979 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
W20251023 20:15:35.653903 23213 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 197) took 3246 ms (client timeout 1000 ms). Trace:
W20251023 20:15:35.714761 23213 rpcz_store.cc:269] 1023 20:15:32.407678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.215389 (+807711us) service_pool.cc:225] Handling call
1023 20:15:35.653862 (+2438473us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":237471,"reported_tablets":228}
I20251023 20:15:35.720602 23786 tablet_bootstrap.cc:492] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.720824 23786 ts_tablet_manager.cc:1403] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.007s
I20251023 20:15:35.721614 23786 raft_consensus.cc:359] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.721838 23786 raft_consensus.cc:385] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.721921 23786 raft_consensus.cc:740] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.722162 23786 consensus_queue.cc:260] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.722579 23211 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 81) took 6365 ms. Trace:
I20251023 20:15:35.722667 23786 ts_tablet_manager.cc:1434] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.722922 23786 tablet_bootstrap.cc:492] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.722692 23211 rpcz_store.cc:276] 1023 20:15:29.357034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.555514 (+6198480us) service_pool.cc:225] Handling call
1023 20:15:35.556632 (+ 1118us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:35.567919 (+ 11287us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:35.722296 (+154377us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:35.722404 (+ 108us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:35.722520 (+ 116us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.622960 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.623083 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.623091 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:35.623094 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:35.623194 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.623208 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.623212 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:35.623291 (+ 79us) tablet.cc:699] Row locks acquired
1023 20:15:35.623293 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:35.623334 (+ 41us) write_op.cc:270] Start()
1023 20:15:35.623372 (+ 38us) write_op.cc:276] Timestamp: P: 1761250535623278 usec, L: 0
1023 20:15:35.623376 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.714629 (+ 91253us) log.cc:844] Serialized 433 byte log entry
1023 20:15:35.715216 (+ 587us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 000000000098330a 00007f899878e682 00007f8999b70ca5 00007f8999b6f7c9 00007f8999b77bf0 00007f8999b779fd 0000000000a880ec 00007f898c95576f 00007f898c958a3b 00007f898c9588b8 0000000000a880ec
1023 20:15:35.721436 (+ 6220us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.721617 (+ 181us) write_op.cc:301] APPLY: starting
1023 20:15:35.721636 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.721689 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.721692 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.721887 (+ 195us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.721923 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:35.721931 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:35.722027 (+ 96us) log.cc:844] Serialized 29 byte log entry
1023 20:15:35.722123 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.722158 (+ 35us) write_op.cc:454] Released schema lock
1023 20:15:35.722165 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":113,"apply.run_cpu_time_us":618,"apply.run_wall_time_us":619,"num_ops":2,"prepare.queue_time_us":54636,"raft.queue_time_us":403,"raft.run_cpu_time_us":487,"raft.run_wall_time_us":486,"replication_time_us":98028,"spinlock_wait_cycles":208969600}]]}
I20251023 20:15:35.724519 23786 tablet_bootstrap.cc:654] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.724944 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:35.725178 23219 catalog_manager.cc:5649] T 72d1dde8b0ac4812869ef233c18e550f P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:35.725745 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:35.726075 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:35.726270 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:35.726487 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:35.727569 25462 raft_consensus.cc:493] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.727689 25462 raft_consensus.cc:515] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.727869 23786 tablet_bootstrap.cc:492] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:35.728005 23786 ts_tablet_manager.cc:1403] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251023 20:15:35.728251 25462 leader_election.cc:290] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.728693 23786 raft_consensus.cc:359] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.728832 23786 raft_consensus.cc:385] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.728871 23786 raft_consensus.cc:740] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.729027 23786 consensus_queue.cc:260] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.729579 23786 ts_tablet_manager.cc:1434] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:35.729892 23786 tablet_bootstrap.cc:492] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:35.731632 23786 tablet_bootstrap.cc:654] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.737672 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9be2a866cf46cfbec8db23a6d3f572" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.737978 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9be2a866cf46cfbec8db23a6d3f572" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.738195 23666 raft_consensus.cc:2468] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.741775 23470 leader_election.cc:304] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.745234 25466 raft_consensus.cc:2804] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.745329 25466 raft_consensus.cc:493] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.745388 25466 raft_consensus.cc:3060] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.745671 25276 raft_consensus.cc:493] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.745810 25276 raft_consensus.cc:515] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.746773 25276 leader_election.cc:290] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:35.747722 23216 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 198) took 2251 ms (client timeout 1000 ms). Trace:
I20251023 20:15:35.747987 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c40164e687545c5b2d2751baaee4571" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.747895 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
W20251023 20:15:35.748562 23604 leader_election.cc:343] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.748157 23216 rpcz_store.cc:269] 1023 20:15:33.483997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.575955 (+ 91958us) service_pool.cc:225] Handling call
1023 20:15:33.631583 (+ 55628us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a05fcfe8 00007f89971042c6 00007f8997103ff5 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:35.735039 (+2103456us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:34.340869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.341017 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.341026 (+ 9us) write_op.cc:435] Acquired schema lock
1023 20:15:34.341029 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:34.341401 (+ 372us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.341424 (+ 23us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.341431 (+ 7us) tablet.cc:683] Acquiring locks for 14 operations
1023 20:15:34.341776 (+ 345us) tablet.cc:699] Row locks acquired
1023 20:15:34.341779 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:34.341826 (+ 47us) write_op.cc:270] Start()
1023 20:15:34.341868 (+ 42us) write_op.cc:276] Timestamp: P: 1761250534341801 usec, L: 0
1023 20:15:34.341873 (+ 5us) op_driver.cc:348] REPLICATION: starting
1023 20:15:34.342137 (+ 264us) log.cc:844] Serialized 5516 byte log entry
1023 20:15:34.346909 (+ 4772us) op_driver.cc:464] REPLICATION: finished
1023 20:15:34.347048 (+ 139us) write_op.cc:301] APPLY: starting
1023 20:15:34.347065 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:34.347156 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:34.347185 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:34.347398 (+ 213us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:34.347443 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
1023 20:15:34.347450 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:34.347583 (+ 133us) log.cc:844] Serialized 101 byte log entry
1023 20:15:34.347661 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.347695 (+ 34us) write_op.cc:454] Released schema lock
1023 20:15:34.347704 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:34.347803 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:34.347814 (+ 11us) write_op.cc:454] Released schema lock
Related trace 'op':
1023 20:15:34.993664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:34.993813 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:34.993821 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:34.993823 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:34.993968 (+ 145us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:34.993984 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:34.993990 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:34.994072 (+ 82us) tablet.cc:699] Row locks acquired
1023 20:15:34.994075 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:34.994125 (+ 50us) write_op.cc:270] Start()
1023 20:15:34.994166 (+ 41us) write_op.cc:276] Timestamp: P: 1761250534994084 usec, L: 0
1023 20:15:34.994169 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.067566 (+ 73397us) log.cc:844] Serialized 356 byte log entry
1023 20:15:35.111218 (+ 43652us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.111733 (+ 515us) write_op.cc:301] APPLY: starting
1023 20:15:35.114243 (+ 2510us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.114289 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.114322 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.114413 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.114959 (+ 546us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:35.114969 (+ 10us) write_op.cc:312] APPLY: finished
1023 20:15:35.115117 (+ 148us) log.cc:844] Serialized 23 byte log entry
1023 20:15:35.115188 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.115210 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:35.115217 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:35.115301 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.115306 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2176,"reported_tablets":224,"spinlock_wait_cycles":44282112,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":743,"apply.run_wall_time_us":742,"num_ops":14,"prepare.queue_time_us":25994,"prepare.run_cpu_time_us":1396,"prepare.run_wall_time_us":1396,"raft.queue_time_us":509,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":322,"replication_time_us":5000}],["op",{"apply.queue_time_us":438,"apply.run_cpu_time_us":576,"apply.run_wall_time_us":3564,"num_ops":1,"prepare.queue_time_us":7091,"prepare.run_cpu_time_us":935,"prepare.run_wall_time_us":74410,"raft.queue_time_us":5979,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":285,"replication_time_us":43859,"thread_start_us":35660,"threads_started":2}]]}
I20251023 20:15:35.750921 25040 consensus_queue.cc:1048] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:35.751434 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c40164e687545c5b2d2751baaee4571" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:35.752024 23605 leader_election.cc:343] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.752135 23605 leader_election.cc:304] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.752533 25276 raft_consensus.cc:2749] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.756181 23670 raft_consensus.cc:2468] T 29c2c976d6a3440394f478a31abc7134 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:35.757157 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 367) took 2231 ms. Trace:
I20251023 20:15:35.757283 25282 rpcz_store.cc:276] 1023 20:15:33.526079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.526123 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:35.475424 (+1949301us) ts_tablet_manager.cc:1151] Deleting tablet 39a62c97e0a3459e91c24c1c2f92389a
1023 20:15:35.475432 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.582148 (+106716us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.688232 (+106084us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.757138 (+ 68906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1943939}
I20251023 20:15:35.757897 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 368) took 2231 ms. Trace:
I20251023 20:15:35.758005 25282 rpcz_store.cc:276] 1023 20:15:33.526499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.526551 (+ 52us) service_pool.cc:225] Handling call
1023 20:15:35.757762 (+2231211us) ts_tablet_manager.cc:1151] Deleting tablet 1e19859a65a144598e1612be89daa3a1
1023 20:15:35.757772 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.757881 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2226232}
I20251023 20:15:35.758457 25282 tablet_replica.cc:333] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:35.758698 25282 raft_consensus.cc:2243] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.758886 25282 raft_consensus.cc:2272] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.760761 23216 master_service.cc:954] ReplaceTablet: received request to replace tablet 2546fa8cb2004463af57f29ba28fdc1d from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:35.748430 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:35.762287 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:35.762594 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:35.762984 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:35.763326 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:35.763504 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:35.763878 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:35.764720 23219 catalog_manager.cc:5649] T e18c1df87ee640148f5e827ee168fea6 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
W20251023 20:15:35.765395 25167 consensus_meta.cc:287] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.631s user 0.002s sys 0.000s
I20251023 20:15:35.765507 25167 raft_consensus.cc:515] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.765863 25462 raft_consensus.cc:493] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.766006 25462 raft_consensus.cc:515] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.766181 25167 leader_election.cc:290] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.766512 25462 leader_election.cc:290] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.770020 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 39a62c97e0a3459e91c24c1c2f92389a (table default.Table_14cd5c04c2b84209a82106f3424a8bd0 [id=f6d53450bf2e46678751ec0bc88efcde]) successfully deleted
W20251023 20:15:35.770290 23204 catalog_manager.cc:4965] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet 1e19859a65a144598e1612be89daa3a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e19859a65a144598e1612be89daa3a1 already in progress: creating tablet
I20251023 20:15:35.770745 25462 raft_consensus.cc:493] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.770869 25462 raft_consensus.cc:515] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.771390 25462 leader_election.cc:290] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.771842 23392 raft_consensus.cc:2468] T 29c2c976d6a3440394f478a31abc7134 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:35.766646 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:35.773200 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:35.774180 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:35.774392 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:35.774852 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:35.775771 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b220ceff0924b26ae8a54dfb8705456" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:35.775925 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dae906f257c346d7a1b28c9052d600b5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.775983 23670 raft_consensus.cc:3060] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.776082 23664 raft_consensus.cc:2468] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.776430 23470 leader_election.cc:304] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.780933 25462 raft_consensus.cc:2804] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.781106 25462 raft_consensus.cc:697] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:35.781381 25462 consensus_queue.cc:237] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.783360 25235 raft_consensus.cc:493] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.783524 25235 raft_consensus.cc:515] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.783826 23470 leader_election.cc:304] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.784116 25167 raft_consensus.cc:2804] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.784188 25167 raft_consensus.cc:493] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.784168 25235 leader_election.cc:290] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.784247 25167 raft_consensus.cc:3060] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.785327 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.785727 23664 raft_consensus.cc:2393] T 8e718cb7205643fd8c3045c3fab30d3e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:35.795042 23787 tablet_bootstrap.cc:492] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.795197 23787 ts_tablet_manager.cc:1403] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.134s user 0.009s sys 0.007s
I20251023 20:15:35.783615 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 82) took 6426 ms. Trace:
I20251023 20:15:35.795949 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4f012469fa4e8e827e485287ad76dd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.796133 23665 raft_consensus.cc:2468] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.796065 23210 rpcz_store.cc:276] 1023 20:15:29.357178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.577446 (+6220268us) service_pool.cc:225] Handling call
1023 20:15:35.646662 (+ 69216us) spinlock_profiling.cc:243] Waited 68.9 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a06027a3 00007f8997106de6 00007f8997106b15 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:35.647575 (+ 913us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:35.648032 (+ 457us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:35.783338 (+135306us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:35.783460 (+ 122us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:35.783563 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.777650 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.777789 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.777793 (+ 4us) write_op.cc:435] Acquired schema lock
1023 20:15:35.777795 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:35.777891 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.777904 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.777909 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:35.777986 (+ 77us) tablet.cc:699] Row locks acquired
1023 20:15:35.777989 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:35.778020 (+ 31us) write_op.cc:270] Start()
1023 20:15:35.778053 (+ 33us) write_op.cc:276] Timestamp: P: 1761250535777996 usec, L: 0
1023 20:15:35.778057 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.778247 (+ 190us) log.cc:844] Serialized 433 byte log entry
1023 20:15:35.781969 (+ 3722us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.782130 (+ 161us) write_op.cc:301] APPLY: starting
1023 20:15:35.782146 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.782194 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.782260 (+ 66us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.782396 (+ 136us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.782434 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:35.782441 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.782526 (+ 85us) log.cc:844] Serialized 29 byte log entry
1023 20:15:35.782604 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.782626 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:35.782633 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:35.793764 (+ 11131us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.793773 (+ 9us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":158504704,"child_traces":[["op",{"apply.queue_time_us":106,"apply.run_cpu_time_us":577,"apply.run_wall_time_us":577,"num_ops":2,"prepare.queue_time_us":129161,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":16095,"raft.queue_time_us":61,"raft.run_cpu_time_us":607,"raft.run_wall_time_us":608,"replication_time_us":3879}]]}
I20251023 20:15:35.796551 23470 leader_election.cc:304] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.796809 25568 raft_consensus.cc:2804] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.796906 25568 raft_consensus.cc:493] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.796964 25568 raft_consensus.cc:3060] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.796990 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_68a2dbd9e4cb46cbab9b4b9e49e64d15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:35.797626 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68a2dbd9e4cb46cbab9b4b9e49e64d15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:35.793401 23213 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_a5e06ba2ba784868978a86ee8e607a2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:35.799302 23213 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5e06ba2ba784868978a86ee8e607a2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:35.804299 23444 heartbeater.cc:344] Connected to a master server at 127.22.127.188:41417
I20251023 20:15:35.804478 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
W20251023 20:15:35.750819 23471 leader_election.cc:343] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.809435 23471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 58623.9us
I20251023 20:15:35.812770 25282 ts_tablet_manager.cc:1916] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:35.813191 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 85) took 6425 ms. Trace:
I20251023 20:15:35.813316 23210 rpcz_store.cc:276] 1023 20:15:29.387691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.796777 (+6409086us) service_pool.cc:225] Handling call
1023 20:15:35.797716 (+ 939us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:35.798158 (+ 442us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:35.812909 (+ 14751us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:35.813033 (+ 124us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:35.813146 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.804353 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.804566 (+ 213us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.804573 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:35.804576 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:35.804774 (+ 198us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.804791 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.804802 (+ 11us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:35.804886 (+ 84us) tablet.cc:699] Row locks acquired
1023 20:15:35.804889 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:35.804931 (+ 42us) write_op.cc:270] Start()
1023 20:15:35.804971 (+ 40us) write_op.cc:276] Timestamp: P: 1761250535804901 usec, L: 0
1023 20:15:35.804974 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.805214 (+ 240us) log.cc:844] Serialized 433 byte log entry
1023 20:15:35.810056 (+ 4842us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.810228 (+ 172us) write_op.cc:301] APPLY: starting
1023 20:15:35.810245 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.810293 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.810317 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.810446 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.810504 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:35.810511 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.810607 (+ 96us) log.cc:844] Serialized 29 byte log entry
1023 20:15:35.810672 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.810694 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:35.810701 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":117,"apply.run_cpu_time_us":539,"apply.run_wall_time_us":539,"num_ops":2,"prepare.queue_time_us":5625,"raft.queue_time_us":44,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":246,"replication_time_us":5051}]]}
I20251023 20:15:35.814867 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e718cb7205643fd8c3045c3fab30d3e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.815595 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b220ceff0924b26ae8a54dfb8705456" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:35.815766 23392 raft_consensus.cc:3060] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.816751 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_39ef492038e649d1b5378fedd9cda522"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:35.817394 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39ef492038e649d1b5378fedd9cda522 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:35.795969 23787 raft_consensus.cc:359] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.818869 23787 raft_consensus.cc:385] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.818848 25040 raft_consensus.cc:493] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.818920 23787 raft_consensus.cc:740] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.818948 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4f012469fa4e8e827e485287ad76dd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.818976 25040 raft_consensus.cc:515] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.819123 23787 consensus_queue.cc:260] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.819613 23787 ts_tablet_manager.cc:1434] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.024s user 0.000s sys 0.000s
I20251023 20:15:35.819613 25040 leader_election.cc:290] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.819931 23787 tablet_bootstrap.cc:492] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.821475 23787 tablet_bootstrap.cc:654] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.822134 25462 raft_consensus.cc:493] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.822283 25462 raft_consensus.cc:515] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.822924 25462 leader_election.cc:290] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:35.824736 23470 leader_election.cc:343] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 39a62c97e0a3459e91c24c1c2f92389a
W20251023 20:15:35.825302 23333 leader_election.cc:343] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 39a62c97e0a3459e91c24c1c2f92389a
I20251023 20:15:35.828171 23213 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 83) took 6470 ms. Trace:
I20251023 20:15:35.828301 23213 rpcz_store.cc:276] 1023 20:15:29.357254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.724125 (+6366871us) service_pool.cc:225] Handling call
1023 20:15:35.793334 (+ 69209us) spinlock_profiling.cc:243] Waited 68.5 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a06027a3 00007f8997106de6 00007f8997106b15 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:35.799388 (+ 6054us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:35.799799 (+ 411us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:35.827926 (+ 28127us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:35.828039 (+ 113us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:35.828129 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.822426 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.822536 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.822541 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:35.822543 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:35.822645 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.822659 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.822664 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:35.822739 (+ 75us) tablet.cc:699] Row locks acquired
1023 20:15:35.822741 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:35.822778 (+ 37us) write_op.cc:270] Start()
1023 20:15:35.822812 (+ 34us) write_op.cc:276] Timestamp: P: 1761250535822749 usec, L: 0
1023 20:15:35.822815 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.822995 (+ 180us) log.cc:844] Serialized 433 byte log entry
1023 20:15:35.826880 (+ 3885us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.827021 (+ 141us) write_op.cc:301] APPLY: starting
1023 20:15:35.827037 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.827082 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.827116 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.827220 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.827253 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:35.827260 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.827336 (+ 76us) log.cc:844] Serialized 29 byte log entry
1023 20:15:35.827392 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.827412 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:35.827418 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":157463424,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":453,"apply.run_wall_time_us":452,"num_ops":2,"prepare.queue_time_us":22195,"raft.queue_time_us":34,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":246,"replication_time_us":4036}]]}
I20251023 20:15:35.828953 23213 master_service.cc:954] ReplaceTablet: received request to replace tablet ac55e14745534a02a2f6a422cab6b149 from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:35.829988 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dae906f257c346d7a1b28c9052d600b5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.830161 23399 raft_consensus.cc:2468] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.832785 23333 leader_election.cc:304] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.833235 25040 raft_consensus.cc:3060] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:35.834252 23471 leader_election.cc:343] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.840824 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39a62c97e0a3459e91c24c1c2f92389a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.841008 23399 raft_consensus.cc:2468] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.843253 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39a62c97e0a3459e91c24c1c2f92389a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.843432 23534 raft_consensus.cc:2468] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.844409 23333 leader_election.cc:304] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.845192 25235 raft_consensus.cc:2804] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.845274 25235 raft_consensus.cc:493] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.845332 25235 raft_consensus.cc:3060] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.847193 23213 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 87) took 6401 ms. Trace:
I20251023 20:15:35.847304 23213 rpcz_store.cc:276] 1023 20:15:29.445719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.828880 (+6383161us) service_pool.cc:225] Handling call
1023 20:15:35.846938 (+ 18058us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a060f4eb 00007f8997113727 00007f8997113455 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:35.847123 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40785152}
I20251023 20:15:35.846504 23471 leader_election.cc:304] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.847750 23213 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_464dc629d50b4995b30ef5c463e9f316" } new_table_name: "default.Table_3eeb50ae897f4006ad63033f2ad0324a" modify_external_catalogs: true
I20251023 20:15:35.848582 25462 raft_consensus.cc:2804] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.848659 25462 raft_consensus.cc:493] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.848714 25462 raft_consensus.cc:3060] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.851351 24297 raft_consensus.cc:493] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bc6522db11e4960a9aad84435371a32)
I20251023 20:15:35.851493 24297 raft_consensus.cc:515] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:35.852012 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:35.852106 24297 leader_election.cc:290] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:35.854104 23333 leader_election.cc:343] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 4b6c3e7169154e3497dd3570dca0d745
I20251023 20:15:35.857532 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:35.858076 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:35.858413 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:35.858762 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:35.858987 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:35.859385 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:35.859477 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 86) took 6459 ms. Trace:
I20251023 20:15:35.859635 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:35.859591 23210 rpcz_store.cc:276] 1023 20:15:29.400305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.813838 (+6413533us) service_pool.cc:225] Handling call
1023 20:15:35.816693 (+ 2855us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a06027a3 00007f8997106de6 00007f8997106b15 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:35.817513 (+ 820us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:35.818130 (+ 617us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:35.859226 (+ 41096us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:35.859341 (+ 115us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:35.859444 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.849509 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.849619 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.849624 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:35.849626 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:35.849717 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.849731 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.849736 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:35.849808 (+ 72us) tablet.cc:699] Row locks acquired
1023 20:15:35.849811 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:35.849841 (+ 30us) write_op.cc:270] Start()
1023 20:15:35.849878 (+ 37us) write_op.cc:276] Timestamp: P: 1761250535849812 usec, L: 0
1023 20:15:35.849882 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.850051 (+ 169us) log.cc:844] Serialized 433 byte log entry
1023 20:15:35.856232 (+ 6181us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.856394 (+ 162us) write_op.cc:301] APPLY: starting
1023 20:15:35.856409 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.856454 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.856487 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.856596 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.856631 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:35.856638 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.856716 (+ 78us) log.cc:844] Serialized 29 byte log entry
1023 20:15:35.856784 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.856804 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:35.856812 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":5933440,"child_traces":[["op",{"apply.queue_time_us":110,"apply.run_cpu_time_us":496,"apply.run_wall_time_us":496,"num_ops":2,"prepare.queue_time_us":30868,"raft.queue_time_us":28,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":270,"replication_time_us":6323}]]}
I20251023 20:15:35.860164 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:35.861361 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:35.864313 23670 raft_consensus.cc:2468] T 1b220ceff0924b26ae8a54dfb8705456 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:35.864846 23531 consensus_meta.cc:287] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.699s user 0.003s sys 0.000s
I20251023 20:15:35.864943 23531 raft_consensus.cc:2468] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 2.
I20251023 20:15:35.865293 24821 raft_consensus.cc:515] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.865967 24821 leader_election.cc:290] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.871745 23470 leader_election.cc:304] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.872035 25554 raft_consensus.cc:2804] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:35.872133 25554 raft_consensus.cc:697] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:35.872350 25554 consensus_queue.cc:237] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.873759 23790 tablet_bootstrap.cc:492] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.873946 23790 ts_tablet_manager.cc:1403] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.230s user 0.021s sys 0.061s
W20251023 20:15:35.875213 23333 leader_election.cc:343] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 01fa5f9c700b458ea147373d78551573
I20251023 20:15:35.874969 23790 raft_consensus.cc:359] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.875487 23790 raft_consensus.cc:385] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.875550 23790 raft_consensus.cc:740] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.875794 23790 consensus_queue.cc:260] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.876046 23787 tablet_bootstrap.cc:492] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.876190 23787 ts_tablet_manager.cc:1403] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.000s
I20251023 20:15:35.876343 23790 ts_tablet_manager.cc:1434] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:35.876626 23790 tablet_bootstrap.cc:492] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.876858 23787 raft_consensus.cc:359] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.877027 23787 raft_consensus.cc:385] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.877099 23787 raft_consensus.cc:740] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.877338 23787 consensus_queue.cc:260] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.878175 23787 ts_tablet_manager.cc:1434] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:35.878856 23790 tablet_bootstrap.cc:654] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.879177 23787 tablet_bootstrap.cc:492] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.881408 23333 leader_election.cc:304] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.881865 24821 raft_consensus.cc:2804] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Leader election won for term 2
I20251023 20:15:35.881990 24821 raft_consensus.cc:697] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 2 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:35.882232 24821 consensus_queue.cc:237] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.886734 23787 tablet_bootstrap.cc:654] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.890090 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01fa5f9c700b458ea147373d78551573" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:35.890295 23531 raft_consensus.cc:3060] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.891448 25489 raft_consensus.cc:493] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.891634 25489 raft_consensus.cc:515] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:35.892107 23333 leader_election.cc:343] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 4b6c3e7169154e3497dd3570dca0d745
I20251023 20:15:35.892212 23333 leader_election.cc:304] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.892939 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ae1a94b011412c851cd96946744fc4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.892974 25172 raft_consensus.cc:2749] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:35.893149 23665 raft_consensus.cc:2468] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.892220 25489 leader_election.cc:290] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.893455 23333 leader_election.cc:304] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:35.893790 25489 raft_consensus.cc:2804] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.893860 25489 raft_consensus.cc:493] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.893918 25489 raft_consensus.cc:3060] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.895875 23219 catalog_manager.cc:5649] T 67e0701159c548c49e9e245820c45028 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:35.896747 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:35.898819 23210 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_934c5152009843bcbf11132bbe4c8ed2" } new_table_name: "default.Table_3c3fd30fc677490da11231e62dd177aa" modify_external_catalogs: true
I20251023 20:15:35.901194 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ae1a94b011412c851cd96946744fc4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.901373 23534 raft_consensus.cc:2468] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.903098 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 169) took 7125 ms. Trace:
I20251023 20:15:35.904867 23219 catalog_manager.cc:5649] T 92c50c4845524163834861438291a3ec P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:35.906658 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:35.907244 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:35.915607 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
I20251023 20:15:35.920382 25276 raft_consensus.cc:493] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.920590 25276 raft_consensus.cc:515] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.921312 25276 leader_election.cc:290] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.922475 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1822fdf1f5f44dd4a742aa3c2f8443cf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:35.923269 23604 leader_election.cc:343] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.925438 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:35.925827 23219 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:35.926407 23787 tablet_bootstrap.cc:492] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.926587 23787 ts_tablet_manager.cc:1403] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20251023 20:15:35.927299 23787 raft_consensus.cc:359] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.927515 23787 raft_consensus.cc:385] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.927603 23787 raft_consensus.cc:740] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.927826 23787 consensus_queue.cc:260] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.928318 23787 ts_tablet_manager.cc:1434] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.928704 23787 tablet_bootstrap.cc:492] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.930102 23787 tablet_bootstrap.cc:654] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.910547 25172 raft_consensus.cc:493] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.924014 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1822fdf1f5f44dd4a742aa3c2f8443cf" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.931468 25172 raft_consensus.cc:515] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:35.932009 23605 leader_election.cc:343] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.932104 25172 leader_election.cc:290] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.932121 23605 leader_election.cc:304] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.932464 25276 raft_consensus.cc:2749] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:35.933108 23333 leader_election.cc:343] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 26a680ad3aed487593bd1b63004dbbfc
I20251023 20:15:35.933450 24780 rpcz_store.cc:276] 1023 20:15:28.778030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.778071 (+ 41us) service_pool.cc:225] Handling call
1023 20:15:35.458826 (+6680755us) ts_tablet_manager.cc:1151] Deleting tablet 419e41d7a37f4b888cb242b089c28790
1023 20:15:35.458845 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.566254 (+107409us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.671617 (+105363us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.903074 (+231457us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6680587}
I20251023 20:15:35.933849 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26a680ad3aed487593bd1b63004dbbfc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.934046 23530 raft_consensus.cc:2468] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.935751 23333 leader_election.cc:304] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:35.937155 25172 raft_consensus.cc:2804] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.937244 25172 raft_consensus.cc:493] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.937310 25172 raft_consensus.cc:3060] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.937345 23790 tablet_bootstrap.cc:492] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.937477 23790 ts_tablet_manager.cc:1403] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.002s
I20251023 20:15:35.937765 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 419e41d7a37f4b888cb242b089c28790 (table default.Table_837a7dbe47ef4dd3ba76317c430a26b1 [id=ba3d13650ed94cc68bcf60e5a8ffd011]) successfully deleted
I20251023 20:15:35.937767 25554 raft_consensus.cc:493] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.937912 25554 raft_consensus.cc:515] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.938469 25554 leader_election.cc:290] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.938409 23790 raft_consensus.cc:359] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.938565 23790 raft_consensus.cc:385] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.938609 23790 raft_consensus.cc:740] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.938918 23790 consensus_queue.cc:260] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.939105 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e343a7d39cf4329b99c9f9ffd8cefb2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.939289 23665 raft_consensus.cc:2468] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.939610 23790 ts_tablet_manager.cc:1434] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:35.939708 23470 leader_election.cc:304] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.939893 23790 tablet_bootstrap.cc:492] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.940028 25554 raft_consensus.cc:2804] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.940099 25554 raft_consensus.cc:493] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.940152 25554 raft_consensus.cc:3060] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.941159 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e343a7d39cf4329b99c9f9ffd8cefb2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.941327 23399 raft_consensus.cc:2468] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.941790 23787 tablet_bootstrap.cc:492] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:35.941956 23787 ts_tablet_manager.cc:1403] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20251023 20:15:35.942222 25566 raft_consensus.cc:493] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.942337 25566 raft_consensus.cc:515] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.942605 23787 raft_consensus.cc:359] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.942746 23787 raft_consensus.cc:385] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.942792 23787 raft_consensus.cc:740] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.942896 25566 leader_election.cc:290] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.942962 23787 consensus_queue.cc:260] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.943410 23787 ts_tablet_manager.cc:1434] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:35.943789 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d71f0cfdb6449fbd74b854c15326d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.944224 23665 raft_consensus.cc:2468] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.944242 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d71f0cfdb6449fbd74b854c15326d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.944380 23399 raft_consensus.cc:2468] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:35.944584 23470 leader_election.cc:304] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:35.944762 23787 tablet_bootstrap.cc:492] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:35.944836 25566 raft_consensus.cc:2804] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.944900 25566 raft_consensus.cc:493] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.944949 25566 raft_consensus.cc:3060] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.941213 24780 tablet_replica.cc:333] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:35.946592 23787 tablet_bootstrap.cc:654] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.946946 23790 tablet_bootstrap.cc:654] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:35.946595 24780 raft_consensus.cc:2243] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.947182 24780 raft_consensus.cc:2272] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.947942 25359 raft_consensus.cc:493] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.948063 25359 raft_consensus.cc:515] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.948616 25359 leader_election.cc:290] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.949570 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.949769 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:35.950350 23604 leader_election.cc:343] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.950752 23605 leader_election.cc:343] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.950847 23605 leader_election.cc:304] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.952127 25359 raft_consensus.cc:2749] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:35.960847 25282 ts_tablet_manager.cc:1929] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:35.960968 25282 log.cc:1199] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/c1f9935f5e4f4daa9df8fe1611ed76c9
I20251023 20:15:35.961455 25282 ts_tablet_manager.cc:1950] T c1f9935f5e4f4daa9df8fe1611ed76c9 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
W20251023 20:15:35.967657 23536 consensus_meta.cc:287] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.697s user 0.003s sys 0.000s
I20251023 20:15:35.967759 23536 raft_consensus.cc:2468] T 92576756ec574ceaa4c81f435cb72301 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:35.968125 23392 raft_consensus.cc:2468] T 1b220ceff0924b26ae8a54dfb8705456 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:35.974016 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 369) took 2447 ms. Trace:
I20251023 20:15:35.974053 24297 raft_consensus.cc:493] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.974164 24297 raft_consensus.cc:515] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.974119 25282 rpcz_store.cc:276] 1023 20:15:33.526743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.526794 (+ 51us) service_pool.cc:225] Handling call
1023 20:15:35.758414 (+2231620us) ts_tablet_manager.cc:1151] Deleting tablet c1f9935f5e4f4daa9df8fe1611ed76c9
1023 20:15:35.758423 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.864705 (+106282us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.960728 (+ 96023us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.973991 (+ 13263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2226858}
I20251023 20:15:35.974316 23790 tablet_bootstrap.cc:492] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.974447 23790 ts_tablet_manager.cc:1403] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20251023 20:15:35.974457 25282 tablet_replica.cc:333] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:35.974696 24297 leader_election.cc:290] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:35.974785 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet c1f9935f5e4f4daa9df8fe1611ed76c9 (table default.Table_c37d72052930485b819533573f3710ac [id=d38b70b1f8e147e2aeadf0fce048c2a1]) successfully deleted
I20251023 20:15:35.975644 25276 raft_consensus.cc:493] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.975720 23790 raft_consensus.cc:359] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.975786 25276 raft_consensus.cc:515] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.975889 23790 raft_consensus.cc:385] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.975934 23790 raft_consensus.cc:740] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.976138 23790 consensus_queue.cc:260] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.976318 25276 leader_election.cc:290] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.976424 25276 raft_consensus.cc:493] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:35.976505 25276 raft_consensus.cc:515] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.976681 23790 ts_tablet_manager.cc:1434] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:35.976969 23790 tablet_bootstrap.cc:492] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.976989 25276 leader_election.cc:290] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:35.977941 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.978335 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d227b45bfcc64d76a51cff347e6e1486" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.979467 23790 tablet_bootstrap.cc:654] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:35.981455 23604 leader_election.cc:343] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:35.981642 23604 leader_election.cc:343] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.982456 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.982612 25282 raft_consensus.cc:2243] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:35.982751 25282 raft_consensus.cc:2272] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:35.982465 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d227b45bfcc64d76a51cff347e6e1486" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:35.983244 25282 ts_tablet_manager.cc:1916] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:35.985435 23605 leader_election.cc:343] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.985530 23605 leader_election.cc:304] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.986191 25359 raft_consensus.cc:2749] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:35.985893 23605 leader_election.cc:343] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:35.986440 23605 leader_election.cc:304] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:35.986745 25359 raft_consensus.cc:2749] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:35.987216 23204 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 26a680ad3aed487593bd1b63004dbbfc (table default.Table_cca4260f97404dc2abc827888a312ee4 [id=d449a30a75d7469d877e700b37b034dc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 26a680ad3aed487593bd1b63004dbbfc consensus state has no leader
I20251023 20:15:35.988993 23790 tablet_bootstrap.cc:492] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:35.989230 23790 ts_tablet_manager.cc:1403] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20251023 20:15:35.989957 23790 raft_consensus.cc:359] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.990170 23790 raft_consensus.cc:385] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:35.990260 23790 raft_consensus.cc:740] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:35.990518 23790 consensus_queue.cc:260] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:35.994061 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5161336598884742ad403fc88c9d9f0b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:35.994231 23670 raft_consensus.cc:2468] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:35.995148 23333 leader_election.cc:304] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:35.995409 24297 raft_consensus.cc:2804] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:35.995492 24297 raft_consensus.cc:493] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:35.995549 24297 raft_consensus.cc:3060] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:35.996342 23790 ts_tablet_manager.cc:1434] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251023 20:15:35.996701 23790 tablet_bootstrap.cc:492] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:35.998301 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5161336598884742ad403fc88c9d9f0b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:35.998463 23536 raft_consensus.cc:2468] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:36.000419 23790 tablet_bootstrap.cc:654] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.006420 23216 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 2546fa8cb2004463af57f29ba28fdc1d
I20251023 20:15:36.006875 23216 catalog_manager.cc:6719] ReplaceTablet: tablet 2546fa8cb2004463af57f29ba28fdc1d deleted and replaced by tablet 6313bf6e463b42bd9990ec50904a3b60
I20251023 20:15:36.006987 23216 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 84) took 6635 ms. Trace:
I20251023 20:15:36.007076 23216 rpcz_store.cc:276] 1023 20:15:29.372013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.760667 (+6388654us) service_pool.cc:225] Handling call
1023 20:15:35.794007 (+ 33340us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a060f4eb 00007f8997113727 00007f8997113455 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:36.006955 (+212948us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.000057 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.000177 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.000182 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.000185 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:36.000314 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.000328 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.000333 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.000405 (+ 72us) tablet.cc:699] Row locks acquired
1023 20:15:36.000407 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:36.000438 (+ 31us) write_op.cc:270] Start()
1023 20:15:36.000470 (+ 32us) write_op.cc:276] Timestamp: P: 1761250536000398 usec, L: 0
1023 20:15:36.001434 (+ 964us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.001666 (+ 232us) log.cc:844] Serialized 631 byte log entry
1023 20:15:36.005470 (+ 3804us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.005643 (+ 173us) write_op.cc:301] APPLY: starting
1023 20:15:36.005659 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.005705 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.005741 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.005858 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.005914 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:36.005922 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:36.005995 (+ 73us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.006051 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.006070 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:36.006076 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":75015936,"child_traces":[["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":490,"apply.run_wall_time_us":489,"num_ops":2,"prepare.queue_time_us":43002,"raft.queue_time_us":38,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":300,"replication_time_us":4002}]]}
I20251023 20:15:36.008831 23499 tablet_service.cc:1552] Processing DeleteTablet for tablet 2546fa8cb2004463af57f29ba28fdc1d with delete_type TABLET_DATA_DELETED (replaced by tablet 6313bf6e463b42bd9990ec50904a3b60) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.009231 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 2546fa8cb2004463af57f29ba28fdc1d with delete_type TABLET_DATA_DELETED (replaced by tablet 6313bf6e463b42bd9990ec50904a3b60) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.012540 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 2546fa8cb2004463af57f29ba28fdc1d with delete_type TABLET_DATA_DELETED (replaced by tablet 6313bf6e463b42bd9990ec50904a3b60) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.013111 24780 ts_tablet_manager.cc:1916] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:36.022610 23790 tablet_bootstrap.cc:492] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.026644 23790 ts_tablet_manager.cc:1403] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20251023 20:15:36.027410 23790 raft_consensus.cc:359] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.027680 23790 raft_consensus.cc:385] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.027781 23790 raft_consensus.cc:740] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.028012 23790 consensus_queue.cc:260] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.032191 23790 ts_tablet_manager.cc:1434] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:36.038255 23790 tablet_bootstrap.cc:492] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.039058 23786 tablet_bootstrap.cc:492] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.039186 23786 ts_tablet_manager.cc:1403] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.309s user 0.062s sys 0.108s
I20251023 20:15:36.039688 23790 tablet_bootstrap.cc:654] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.039872 23786 raft_consensus.cc:359] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.040038 23786 raft_consensus.cc:385] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.040100 23786 raft_consensus.cc:740] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.040272 23786 consensus_queue.cc:260] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.040848 23786 ts_tablet_manager.cc:1434] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.041193 23786 tablet_bootstrap.cc:492] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.042614 23786 tablet_bootstrap.cc:654] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.046250 23216 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_b33239cad93e4f27aee7c9dbe7fab6d9" } new_table_name: "default.Table_7366a659fef84d17925d61af9a4d456c" modify_external_catalogs: true
I20251023 20:15:36.050487 23786 tablet_bootstrap.cc:492] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.051043 23786 ts_tablet_manager.cc:1403] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20251023 20:15:36.051837 23786 raft_consensus.cc:359] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.052771 23786 raft_consensus.cc:385] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.052950 23786 raft_consensus.cc:740] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.054816 23786 consensus_queue.cc:260] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.058903 23790 tablet_bootstrap.cc:492] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.059125 23790 ts_tablet_manager.cc:1403] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20251023 20:15:36.059896 23790 raft_consensus.cc:359] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.060151 23790 raft_consensus.cc:385] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.060238 23790 raft_consensus.cc:740] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.060498 23790 consensus_queue.cc:260] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.061206 23790 ts_tablet_manager.cc:1434] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.061544 23790 tablet_bootstrap.cc:492] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.063087 23790 tablet_bootstrap.cc:654] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.067157 25040 raft_consensus.cc:2749] T 8e718cb7205643fd8c3045c3fab30d3e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:36.073277 25464 consensus_meta.cc:287] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.738s user 0.003s sys 0.000s
I20251023 20:15:36.073405 25464 raft_consensus.cc:515] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.074008 25464 leader_election.cc:290] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.076669 25359 raft_consensus.cc:493] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.076821 25359 raft_consensus.cc:515] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.077438 23786 ts_tablet_manager.cc:1434] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.026s user 0.004s sys 0.008s
I20251023 20:15:36.077471 25359 leader_election.cc:290] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.077745 23786 tablet_bootstrap.cc:492] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.078739 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.079087 23786 tablet_bootstrap.cc:654] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:36.079536 23605 leader_election.cc:343] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.084362 25464 raft_consensus.cc:493] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.084487 25464 raft_consensus.cc:515] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.084668 23786 tablet_bootstrap.cc:492] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.084789 23786 ts_tablet_manager.cc:1403] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:36.085047 25464 leader_election.cc:290] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.086540 23786 raft_consensus.cc:359] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.086968 23786 raft_consensus.cc:385] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.087141 23786 raft_consensus.cc:740] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.087428 23786 consensus_queue.cc:260] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.088011 23790 tablet_bootstrap.cc:492] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.088188 23790 ts_tablet_manager.cc:1403] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20251023 20:15:36.088246 23786 ts_tablet_manager.cc:1434] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:36.089176 23786 tablet_bootstrap.cc:492] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.089682 23790 raft_consensus.cc:359] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.089936 23790 raft_consensus.cc:385] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.090004 23790 raft_consensus.cc:740] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.090233 23790 consensus_queue.cc:260] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.090756 23790 ts_tablet_manager.cc:1434] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.091090 23790 tablet_bootstrap.cc:492] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.091723 23786 tablet_bootstrap.cc:654] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.092499 23790 tablet_bootstrap.cc:654] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.093259 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a744214a24cc4bf28654ffa417ce0d3d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:36.093430 23392 raft_consensus.cc:3060] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.093916 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3befdfc85bca42bb816e11d1dd832c8c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.100653 23786 tablet_bootstrap.cc:492] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.100636 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a744214a24cc4bf28654ffa417ce0d3d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.100837 23670 raft_consensus.cc:3060] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.100862 23786 ts_tablet_manager.cc:1403] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20251023 20:15:36.101528 23786 raft_consensus.cc:359] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.101682 23786 raft_consensus.cc:385] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.101724 23786 raft_consensus.cc:740] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.101883 23786 consensus_queue.cc:260] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.102315 23786 ts_tablet_manager.cc:1434] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:36.102574 23786 tablet_bootstrap.cc:492] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.103912 23786 tablet_bootstrap.cc:654] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.106191 25359 raft_consensus.cc:493] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.106333 25359 raft_consensus.cc:515] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.106935 25359 leader_election.cc:290] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.108335 23499 tablet_service.cc:1505] Processing CreateTablet for tablet 1705a87bfa7e4f7caa00b4666fd0514a (DEFAULT_TABLE table=default.Table_c1f42ba6bf654a09ade937991a5f85c0 [id=85e0bb2abec640a28b1cc1806b45e52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.108752 23499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1705a87bfa7e4f7caa00b4666fd0514a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.109364 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 1705a87bfa7e4f7caa00b4666fd0514a (DEFAULT_TABLE table=default.Table_c1f42ba6bf654a09ade937991a5f85c0 [id=85e0bb2abec640a28b1cc1806b45e52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.111935 23786 tablet_bootstrap.cc:492] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.112085 23786 ts_tablet_manager.cc:1403] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20251023 20:15:36.112771 23786 raft_consensus.cc:359] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.112949 23786 raft_consensus.cc:385] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.113013 23786 raft_consensus.cc:740] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.113243 23786 consensus_queue.cc:260] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.113449 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 1705a87bfa7e4f7caa00b4666fd0514a (DEFAULT_TABLE table=default.Table_c1f42ba6bf654a09ade937991a5f85c0 [id=85e0bb2abec640a28b1cc1806b45e52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.113745 23786 ts_tablet_manager.cc:1434] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:36.114015 23786 tablet_bootstrap.cc:492] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.115764 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3befdfc85bca42bb816e11d1dd832c8c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.115881 23787 tablet_bootstrap.cc:492] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.115939 23665 raft_consensus.cc:2468] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:36.115990 23787 ts_tablet_manager.cc:1403] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.171s user 0.013s sys 0.053s
I20251023 20:15:36.116608 23787 raft_consensus.cc:359] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.116758 23787 raft_consensus.cc:385] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.116806 23787 raft_consensus.cc:740] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.116981 23787 consensus_queue.cc:260] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.117180 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.117472 23787 ts_tablet_manager.cc:1434] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251023 20:15:36.117519 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e5dc42c01a4d5e9661f399f793f68f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.117756 23787 tablet_bootstrap.cc:492] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.119048 23787 tablet_bootstrap.cc:654] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.115312 23786 tablet_bootstrap.cc:654] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.127295 25359 raft_consensus.cc:493] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.127458 25359 raft_consensus.cc:515] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.128070 25359 leader_election.cc:290] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.128894 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.137986 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e5dc42c01a4d5e9661f399f793f68f" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.138188 23605 leader_election.cc:343] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:36.138487 23605 leader_election.cc:343] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.139878 23790 tablet_bootstrap.cc:492] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.140332 23790 ts_tablet_manager.cc:1403] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20251023 20:15:36.140636 23787 tablet_bootstrap.cc:492] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.140797 23787 ts_tablet_manager.cc:1403] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.010s
I20251023 20:15:36.141276 23790 raft_consensus.cc:359] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.141512 23790 raft_consensus.cc:385] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.141503 23787 raft_consensus.cc:359] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.141602 23790 raft_consensus.cc:740] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.141695 23787 raft_consensus.cc:385] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.141769 23787 raft_consensus.cc:740] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.141892 23790 consensus_queue.cc:260] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.142000 23787 consensus_queue.cc:260] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.145722 23787 ts_tablet_manager.cc:1434] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251023 20:15:36.146034 23787 tablet_bootstrap.cc:492] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.146750 25359 raft_consensus.cc:493] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.146873 25359 raft_consensus.cc:515] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.147439 25359 leader_election.cc:290] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.147897 23787 tablet_bootstrap.cc:654] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.152136 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.153018 23605 leader_election.cc:343] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.155169 23211 catalog_manager.cc:5649] T 882f7bf08e2f4b5382702c3d6eaf201e P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } }
I20251023 20:15:36.156807 23787 tablet_bootstrap.cc:492] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.156997 23787 ts_tablet_manager.cc:1403] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20251023 20:15:36.157753 23787 raft_consensus.cc:359] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.157950 23787 raft_consensus.cc:385] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.158032 23787 raft_consensus.cc:740] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.158258 23787 consensus_queue.cc:260] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.158766 23787 ts_tablet_manager.cc:1434] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:36.164608 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1705a87bfa7e4f7caa00b4666fd0514a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.165436 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:36.165704 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:36.165961 23211 catalog_manager.cc:5649] T 9b766ff2dff34a8f80f70591dba54017 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } }
I20251023 20:15:36.166265 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:36.166492 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
I20251023 20:15:36.166854 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 757d41e027ba461ea051b38a2d685407 (table default.Table_d78ebb55a1754b5bb096fab938774ce1 [id=dd33d0beee524dbbb6f9c73b46ca7912]). Expected version 1 got 0
I20251023 20:15:36.167078 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:36.167308 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 124703edbf08408a9ca69732674c0ad3 (table default.Table_c07286b67c934b439529460275aa2631 [id=98abf33cd7324a51902de8bbf62de4b6]). Expected version 2 got 0
I20251023 20:15:36.168283 25235 raft_consensus.cc:515] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.168924 25235 leader_election.cc:290] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:36.169224 25465 consensus_meta.cc:287] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.787s user 0.000s sys 0.003s
I20251023 20:15:36.169332 25465 raft_consensus.cc:515] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.169926 25465 leader_election.cc:290] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.171231 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:36.171645 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:36.171789 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:36.171923 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:36.172122 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:36.172385 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:36.172581 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:36.172654 23790 ts_tablet_manager.cc:1434] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251023 20:15:36.172901 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:36.172996 23790 tablet_bootstrap.cc:492] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.173183 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:36.173321 25599 raft_consensus.cc:493] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.173446 25599 raft_consensus.cc:515] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.173975 25599 leader_election.cc:290] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.174468 25596 raft_consensus.cc:493] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.174615 25596 raft_consensus.cc:515] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.175069 25596 leader_election.cc:290] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:36.175369 23471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 53565.5us
W20251023 20:15:36.175560 23471 leader_election.cc:343] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.177327 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39a62c97e0a3459e91c24c1c2f92389a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:36.178122 23211 catalog_manager.cc:5649] T f03df77c344b4736a30725e4d9f72a35 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } }
I20251023 20:15:36.178627 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:36.179291 25600 consensus_queue.cc:1048] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.179504 25235 raft_consensus.cc:493] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.179634 25235 raft_consensus.cc:515] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.180190 25235 leader_election.cc:290] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.159072 23787 tablet_bootstrap.cc:492] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.164306 23670 raft_consensus.cc:2468] T a744214a24cc4bf28654ffa417ce0d3d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.170316 25282 ts_tablet_manager.cc:1929] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:36.181253 25282 log.cc:1199] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/370a790a053446b2b06314c7db242270
I20251023 20:15:36.185957 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
W20251023 20:15:36.186345 23333 leader_election.cc:343] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: d9f73477a06542c0b215ae86716cdc4e
I20251023 20:15:36.186697 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1705a87bfa7e4f7caa00b4666fd0514a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.187232 25282 ts_tablet_manager.cc:1950] T 370a790a053446b2b06314c7db242270 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:36.189277 23790 tablet_bootstrap.cc:654] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.190672 23470 leader_election.cc:304] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:36.191910 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:36.192453 25599 raft_consensus.cc:2804] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.192562 25599 raft_consensus.cc:493] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.192646 25599 raft_consensus.cc:3060] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.193063 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:36.200444 23787 tablet_bootstrap.cc:654] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.200850 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:36.201753 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:36.204591 23470 leader_election.cc:304] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.206602 25465 raft_consensus.cc:2804] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.206714 25465 raft_consensus.cc:697] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:36.206959 25465 consensus_queue.cc:237] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.212405 23787 tablet_bootstrap.cc:492] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.212618 23787 ts_tablet_manager.cc:1403] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.004s
I20251023 20:15:36.213851 23787 raft_consensus.cc:359] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.214073 23787 raft_consensus.cc:385] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.214166 23787 raft_consensus.cc:740] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.214432 23787 consensus_queue.cc:260] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.217356 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]). Expected version 1 got 0
W20251023 20:15:36.218310 23604 leader_election.cc:343] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.218421 23604 leader_election.cc:304] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.218801 25359 raft_consensus.cc:2749] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:36.219398 23604 leader_election.cc:343] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.219494 23604 leader_election.cc:304] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.219812 25359 raft_consensus.cc:2749] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.219945 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]). Expected version 1 got 0
I20251023 20:15:36.220748 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b593b020b34a969f77d2f68178e89d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.220904 23670 raft_consensus.cc:3060] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.221140 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:36.221426 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:36.223153 23787 ts_tablet_manager.cc:1434] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251023 20:15:36.223510 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:36.224049 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:36.224362 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.224516 23665 raft_consensus.cc:2468] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:36.189733 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b593b020b34a969f77d2f68178e89d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:36.230481 23398 raft_consensus.cc:3060] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.190124 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.191056 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "370a790a053446b2b06314c7db242270" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.230973 23393 raft_consensus.cc:2468] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:36.231010 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:36.231982 23787 tablet_bootstrap.cc:492] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:36.217890 23333 leader_election.cc:343] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 39a62c97e0a3459e91c24c1c2f92389a
W20251023 20:15:36.232398 23333 consensus_peers.cc:597] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e194d03a6aa946fbb2c79d91b196440b. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:36.232606 23333 leader_election.cc:343] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d9f73477a06542c0b215ae86716cdc4e
I20251023 20:15:36.232659 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:36.232748 23333 leader_election.cc:304] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.233044 25235 raft_consensus.cc:2749] T d9f73477a06542c0b215ae86716cdc4e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.234010 23787 tablet_bootstrap.cc:654] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.232952 23211 catalog_manager.cc:5649] T 94cd8880a78c4b4eb0c19ba5944b12f2 P 9bc6522db11e4960a9aad84435371a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130). New cstate: current_term: 1 leader_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } }
I20251023 20:15:36.237684 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 92c50c4845524163834861438291a3ec (table default.Table_5ba404e4eee84ed4bf30bf1be1236aa5 [id=a8bd87490f4d46b7ad7df86a5ad6e11e]). Expected version 1 got 0
I20251023 20:15:36.237959 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:36.238061 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.238682 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.239141 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
W20251023 20:15:36.239835 23604 leader_election.cc:343] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.239956 23604 leader_election.cc:304] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.240303 25359 raft_consensus.cc:2749] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:36.240680 23604 leader_election.cc:343] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.240772 23604 leader_election.cc:304] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.241122 25359 raft_consensus.cc:2749] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.241573 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:36.242256 23470 leader_election.cc:304] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.242439 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:36.245720 23787 tablet_bootstrap.cc:492] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.245858 23787 ts_tablet_manager.cc:1403] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20251023 20:15:36.246125 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:36.246532 23787 raft_consensus.cc:359] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.246685 23787 raft_consensus.cc:385] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.246735 23787 raft_consensus.cc:740] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.246960 23787 consensus_queue.cc:260] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.247193 23211 catalog_manager.cc:5741] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
W20251023 20:15:36.247889 23210 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa0a46ffa626447eab4cdb956e798005 consensus state has no leader
I20251023 20:15:36.248061 23210 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 89) took 5627 ms. Trace:
I20251023 20:15:36.248150 23210 rpcz_store.cc:276] 1023 20:15:30.620218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.860273 (+5240055us) service_pool.cc:225] Handling call
1023 20:15:35.898764 (+ 38491us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0604a36 00007f899710a766 00007f899710a495 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:35.899029 (+ 265us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:35.899240 (+ 211us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:35.899452 (+ 212us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:35.899532 (+ 80us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:35.935108 (+ 35576us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:36.248028 (+312920us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.926131 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.926254 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.926258 (+ 4us) write_op.cc:435] Acquired schema lock
1023 20:15:35.926260 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:35.927226 (+ 966us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.927272 (+ 46us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.927278 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:35.927353 (+ 75us) tablet.cc:699] Row locks acquired
1023 20:15:35.927356 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:35.927390 (+ 34us) write_op.cc:270] Start()
1023 20:15:35.927431 (+ 41us) write_op.cc:276] Timestamp: P: 1761250535927355 usec, L: 0
1023 20:15:35.927434 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.928068 (+ 634us) log.cc:844] Serialized 405 byte log entry
1023 20:15:35.931684 (+ 3616us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.931805 (+ 121us) write_op.cc:301] APPLY: starting
1023 20:15:35.931822 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.931864 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.931895 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.931953 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.931984 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:35.931991 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.932075 (+ 84us) log.cc:844] Serialized 23 byte log entry
1023 20:15:35.932146 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.932165 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:35.932172 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:35.932258 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.932263 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":87944448,"child_traces":[["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":432,"apply.run_wall_time_us":432,"num_ops":1,"prepare.queue_time_us":26116,"prepare.run_cpu_time_us":808,"prepare.run_wall_time_us":5193,"raft.queue_time_us":1355,"raft.run_cpu_time_us":486,"raft.run_wall_time_us":487,"replication_time_us":4221}]]}
I20251023 20:15:36.249046 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_3448a0da3d5b4fffa69b2f473d4416b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.249637 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3448a0da3d5b4fffa69b2f473d4416b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:36.253876 23471 leader_election.cc:343] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.255334 23790 tablet_bootstrap.cc:492] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.255522 23790 ts_tablet_manager.cc:1403] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.000s
I20251023 20:15:36.255596 23787 ts_tablet_manager.cc:1434] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251023 20:15:36.256043 23471 leader_election.cc:304] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.258148 25596 raft_consensus.cc:2804] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.258227 25596 raft_consensus.cc:493] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.258278 25596 raft_consensus.cc:3060] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.258803 25464 raft_consensus.cc:2804] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.258889 25464 raft_consensus.cc:493] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.258952 25464 raft_consensus.cc:3060] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.260322 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 370) took 2733 ms. Trace:
I20251023 20:15:36.260409 25282 rpcz_store.cc:276] 1023 20:15:33.526984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.527051 (+ 67us) service_pool.cc:225] Handling call
1023 20:15:35.974410 (+2447359us) ts_tablet_manager.cc:1151] Deleting tablet 370a790a053446b2b06314c7db242270
1023 20:15:35.974421 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.079335 (+104914us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.170220 (+ 90885us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.260305 (+ 90085us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7945,"tablet-delete.queue_time_us":2442814}
I20251023 20:15:36.261935 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 91) took 4955 ms. Trace:
I20251023 20:15:36.262044 23210 rpcz_store.cc:276] 1023 20:15:31.307065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.248840 (+4941775us) service_pool.cc:225] Handling call
1023 20:15:36.249736 (+ 896us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:36.250185 (+ 449us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:36.261592 (+ 11407us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:36.261693 (+ 101us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:36.261897 (+ 204us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.251356 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.251468 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.251475 (+ 7us) write_op.cc:435] Acquired schema lock
1023 20:15:36.251478 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:36.251581 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.251595 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.251600 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.251676 (+ 76us) tablet.cc:699] Row locks acquired
1023 20:15:36.251679 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.251714 (+ 35us) write_op.cc:270] Start()
1023 20:15:36.251748 (+ 34us) write_op.cc:276] Timestamp: P: 1761250536251697 usec, L: 0
1023 20:15:36.251752 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.251925 (+ 173us) log.cc:844] Serialized 433 byte log entry
1023 20:15:36.257660 (+ 5735us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.257816 (+ 156us) write_op.cc:301] APPLY: starting
1023 20:15:36.257832 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.257896 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.257956 (+ 60us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.258095 (+ 139us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.258129 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:36.258136 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.258212 (+ 76us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.258271 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.258290 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:36.258296 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":541,"apply.run_wall_time_us":542,"num_ops":2,"prepare.queue_time_us":663,"raft.queue_time_us":430,"raft.run_cpu_time_us":560,"raft.run_wall_time_us":560,"replication_time_us":5880}]]}
I20251023 20:15:36.263797 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_877d37beafd74b67a50c835a42249829"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.264405 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_877d37beafd74b67a50c835a42249829 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.264442 23786 tablet_bootstrap.cc:492] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.264580 23786 ts_tablet_manager.cc:1403] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.151s user 0.005s sys 0.029s
W20251023 20:15:36.265239 25262 consensus_meta.cc:287] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.782s user 0.000s sys 0.003s
I20251023 20:15:36.265338 25262 raft_consensus.cc:515] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.262562 25594 raft_consensus.cc:493] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.265900 25262 leader_election.cc:290] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.265993 25594 raft_consensus.cc:515] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.266584 25594 leader_election.cc:290] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.267328 23790 raft_consensus.cc:359] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.267453 23790 raft_consensus.cc:385] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.267486 23790 raft_consensus.cc:740] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.267632 23790 consensus_queue.cc:260] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.268132 23790 ts_tablet_manager.cc:1434] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:36.270478 25489 raft_consensus.cc:515] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.271023 25489 leader_election.cc:290] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.271492 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 370a790a053446b2b06314c7db242270 (table default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]) successfully deleted
I20251023 20:15:36.272048 25282 tablet_replica.cc:333] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:36.272267 25282 raft_consensus.cc:2243] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:36.272424 25282 raft_consensus.cc:2272] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:36.272961 25282 ts_tablet_manager.cc:1916] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:36.273424 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "370a790a053446b2b06314c7db242270" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.274855 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53f33020319d48c791e1156c67b55be6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.275004 23665 raft_consensus.cc:3060] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.275857 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ae1a94b011412c851cd96946744fc4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.276005 23666 raft_consensus.cc:3060] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.276407 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 92) took 4967 ms. Trace:
I20251023 20:15:36.276520 23210 rpcz_store.cc:276] 1023 20:15:31.310693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.263049 (+4952356us) service_pool.cc:225] Handling call
1023 20:15:36.264495 (+ 1446us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:36.264960 (+ 465us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:36.276161 (+ 11201us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:36.276269 (+ 108us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:36.276365 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.265595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.265694 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.265700 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:36.265702 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.265785 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.265798 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.265803 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.265892 (+ 89us) tablet.cc:699] Row locks acquired
1023 20:15:36.265895 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.265927 (+ 32us) write_op.cc:270] Start()
1023 20:15:36.265959 (+ 32us) write_op.cc:276] Timestamp: P: 1761250536265911 usec, L: 0
1023 20:15:36.265963 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.266226 (+ 263us) log.cc:844] Serialized 433 byte log entry
1023 20:15:36.270276 (+ 4050us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.270436 (+ 160us) write_op.cc:301] APPLY: starting
1023 20:15:36.270453 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.270501 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.270547 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.270667 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.270702 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:36.270709 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.270789 (+ 80us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.270853 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.270873 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:36.270879 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1061760,"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":523,"num_ops":2,"prepare.queue_time_us":145,"raft.queue_time_us":29,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":240,"replication_time_us":4284}]]}
I20251023 20:15:36.277968 23670 raft_consensus.cc:2468] T 27b593b020b34a969f77d2f68178e89d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:36.279428 23333 consensus_peers.cc:597] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00ec606cbf834ccd8762e034b520e58e. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:36.279636 23333 leader_election.cc:343] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 857da3642f0846ce92aaed0491e89644
I20251023 20:15:36.281841 23786 raft_consensus.cc:359] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.282117 23786 raft_consensus.cc:385] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.282204 23786 raft_consensus.cc:740] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.282390 23786 consensus_queue.cc:260] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.282909 23786 ts_tablet_manager.cc:1434] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251023 20:15:36.283205 23786 tablet_bootstrap.cc:492] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.285897 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_c2d9a9353b6944648cb319a6ea0cf2f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.287022 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2d9a9353b6944648cb319a6ea0cf2f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.287375 23790 tablet_bootstrap.cc:492] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.289362 23787 tablet_bootstrap.cc:492] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.291028 23787 tablet_bootstrap.cc:654] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.291383 24780 ts_tablet_manager.cc:1929] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:36.291532 24780 log.cc:1199] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/951b2276d3324c2d999c8d7fb97767bc
I20251023 20:15:36.291898 24780 ts_tablet_manager.cc:1950] T 951b2276d3324c2d999c8d7fb97767bc P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:36.294430 23786 tablet_bootstrap.cc:654] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.300351 23787 tablet_bootstrap.cc:492] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.300535 23787 ts_tablet_manager.cc:1403] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20251023 20:15:36.301713 23787 raft_consensus.cc:359] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.301929 23787 raft_consensus.cc:385] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.302016 23787 raft_consensus.cc:740] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.302249 23787 consensus_queue.cc:260] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.297890 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "857da3642f0846ce92aaed0491e89644" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.298185 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ae1a94b011412c851cd96946744fc4" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:36.304100 23528 raft_consensus.cc:3060] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:36.305274 23215 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 766882a030b446eb9406b3e32d903ee5 consensus state has no leader
I20251023 20:15:36.305444 23215 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 80) took 7281 ms. Trace:
I20251023 20:15:36.305547 23215 rpcz_store.cc:276] 1023 20:15:29.024355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.553931 (+6529576us) service_pool.cc:225] Handling call
1023 20:15:35.554315 (+ 384us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:35.566447 (+ 12132us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:35.566641 (+ 194us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:35.566741 (+ 100us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:35.574706 (+ 7965us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:36.305415 (+730709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.567383 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.567478 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.567483 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:35.567486 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:35.567572 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.567587 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.567591 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:35.567657 (+ 66us) tablet.cc:699] Row locks acquired
1023 20:15:35.567659 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:35.567691 (+ 32us) write_op.cc:270] Start()
1023 20:15:35.567720 (+ 29us) write_op.cc:276] Timestamp: P: 1761250535567643 usec, L: 0
1023 20:15:35.567724 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.567931 (+ 207us) log.cc:844] Serialized 405 byte log entry
1023 20:15:35.573458 (+ 5527us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.574139 (+ 681us) write_op.cc:301] APPLY: starting
1023 20:15:35.574149 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.574178 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.574209 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.574239 (+ 30us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.574281 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:35.574289 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:35.574338 (+ 49us) log.cc:844] Serialized 23 byte log entry
1023 20:15:35.574373 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.574386 (+ 13us) write_op.cc:454] Released schema lock
1023 20:15:35.574391 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":653,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":300,"num_ops":1,"prepare.queue_time_us":216,"prepare.run_cpu_time_us":10228,"prepare.run_wall_time_us":55525,"raft.queue_time_us":22702,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":206,"replication_time_us":5719,"thread_start_us":127457,"threads_started":2}]]}
I20251023 20:15:36.306298 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_21c86d62c5c840098934de32762c13cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:36.306648 25600 consensus_queue.cc:1048] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:36.306887 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21c86d62c5c840098934de32762c13cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.309473 23210 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 93) took 4806 ms. Trace:
I20251023 20:15:36.310276 23787 ts_tablet_manager.cc:1434] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20251023 20:15:36.311662 23787 tablet_bootstrap.cc:492] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.318747 23786 tablet_bootstrap.cc:492] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.318933 23786 ts_tablet_manager.cc:1403] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20251023 20:15:36.319641 23786 raft_consensus.cc:359] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.319836 23786 raft_consensus.cc:385] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.319916 23786 raft_consensus.cc:740] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.320122 23786 consensus_queue.cc:260] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.320641 23786 ts_tablet_manager.cc:1434] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.320904 23786 tablet_bootstrap.cc:492] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.309731 23214 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 21a508e875c449c5831968562263f1bf
I20251023 20:15:36.323408 23214 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 78) took 7815 ms. Trace:
I20251023 20:15:36.323505 23786 tablet_bootstrap.cc:654] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.323516 23214 rpcz_store.cc:276] 1023 20:15:28.507557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.532169 (+7024612us) service_pool.cc:225] Handling call
1023 20:15:35.537192 (+ 5023us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:35.564922 (+ 27730us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:35.565005 (+ 83us) catalog_manager.cc:2824] Locking tablets
1023 20:15:35.956317 (+391312us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:36.052359 (+ 96042us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:36.052443 (+ 84us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:36.309690 (+257247us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:36.323341 (+ 13651us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.047665 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.047774 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.047779 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.047781 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.047896 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.047909 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.047914 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.048011 (+ 97us) tablet.cc:699] Row locks acquired
1023 20:15:36.048014 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.048044 (+ 30us) write_op.cc:270] Start()
1023 20:15:36.048074 (+ 30us) write_op.cc:276] Timestamp: P: 1761250536048001 usec, L: 0
1023 20:15:36.048078 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.048261 (+ 183us) log.cc:844] Serialized 768 byte log entry
1023 20:15:36.051714 (+ 3453us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.051863 (+ 149us) write_op.cc:301] APPLY: starting
1023 20:15:36.051877 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.051923 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.051948 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.052014 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.052048 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:36.052056 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:36.052140 (+ 84us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.052200 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.052219 (+ 19us) write_op.cc:454] Released schema lock
1023 20:15:36.052226 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:36.094304 (+ 42078us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.094316 (+ 12us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":9088,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":419,"apply.run_wall_time_us":419,"num_ops":2,"prepare.queue_time_us":90605,"prepare.run_cpu_time_us":772,"prepare.run_wall_time_us":46597,"raft.queue_time_us":77,"raft.run_cpu_time_us":641,"raft.run_wall_time_us":640,"replication_time_us":3608}]]}
I20251023 20:15:36.324476 23214 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_572a1f2b4ae0442ba59eb601fffb021c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.325134 23214 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_572a1f2b4ae0442ba59eb601fffb021c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251023 20:15:36.325219 23333 leader_election.cc:343] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.325340 23333 leader_election.cc:304] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.325539 25489 raft_consensus.cc:2749] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.327003 25489 consensus_queue.cc:1048] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.327191 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a508e875c449c5831968562263f1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:35 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.327754 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a508e875c449c5831968562263f1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:35 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.317167 23218 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f] alter complete (version 2)
I20251023 20:15:36.328265 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "53f33020319d48c791e1156c67b55be6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:36.328496 23396 raft_consensus.cc:3060] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:36.330253 23470 leader_election.cc:343] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: Tablet not RUNNING: SHUTDOWN
I20251023 20:15:36.330622 23470 leader_election.cc:304] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.330981 25600 raft_consensus.cc:2804] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.331075 25600 raft_consensus.cc:697] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:36.331284 25600 consensus_queue.cc:237] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:36.321328 23213 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e718cb7205643fd8c3045c3fab30d3e consensus state has no leader
I20251023 20:15:36.333411 23213 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 88) took 5717 ms. Trace:
I20251023 20:15:36.333513 23213 rpcz_store.cc:276] 1023 20:15:30.616040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.847566 (+5231526us) service_pool.cc:225] Handling call
1023 20:15:35.847979 (+ 413us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:35.848181 (+ 202us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:35.848371 (+ 190us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:35.848453 (+ 82us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:35.915469 (+ 67016us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:36.333357 (+417888us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.900153 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.900246 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.900250 (+ 4us) write_op.cc:435] Acquired schema lock
1023 20:15:35.900252 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:35.900341 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.900352 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.900357 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:35.900412 (+ 55us) tablet.cc:699] Row locks acquired
1023 20:15:35.900414 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:35.900446 (+ 32us) write_op.cc:270] Start()
1023 20:15:35.900475 (+ 29us) write_op.cc:276] Timestamp: P: 1761250535900412 usec, L: 0
1023 20:15:35.900479 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.900638 (+ 159us) log.cc:844] Serialized 405 byte log entry
1023 20:15:35.914659 (+ 14021us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.914835 (+ 176us) write_op.cc:301] APPLY: starting
1023 20:15:35.914853 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.914897 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.914923 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.914975 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.915007 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:35.915015 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:35.915114 (+ 99us) log.cc:844] Serialized 23 byte log entry
1023 20:15:35.915179 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.915201 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:35.915209 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:35.926026 (+ 10817us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.926033 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":435,"num_ops":1,"prepare.queue_time_us":51191,"prepare.run_cpu_time_us":669,"prepare.run_wall_time_us":25858,"raft.queue_time_us":8745,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":355,"replication_time_us":14150}]]}
I20251023 20:15:36.334165 23213 master_service.cc:954] ReplaceTablet: received request to replace tablet 98a6a3425a49403e99c8ade114166e46 from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:36.321954 23216 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 90) took 5605 ms. Trace:
I20251023 20:15:36.322520 23217 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet a1efa4fcaeb049b484ca59b12025ce9a
I20251023 20:15:36.335027 23216 rpcz_store.cc:276] 1023 20:15:30.716134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.007578 (+5291444us) service_pool.cc:225] Handling call
1023 20:15:36.046196 (+ 38618us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0604a36 00007f899710a766 00007f899710a495 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:36.046451 (+ 255us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:36.046641 (+ 190us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:36.046837 (+ 196us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:36.046911 (+ 74us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:36.104617 (+ 57706us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:36.321896 (+217279us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.095231 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.095336 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.095341 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.095344 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:36.095424 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.095433 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.095437 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:36.095487 (+ 50us) tablet.cc:699] Row locks acquired
1023 20:15:36.095490 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.095515 (+ 25us) write_op.cc:270] Start()
1023 20:15:36.095544 (+ 29us) write_op.cc:276] Timestamp: P: 1761250536095469 usec, L: 0
1023 20:15:36.095547 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.095681 (+ 134us) log.cc:844] Serialized 405 byte log entry
1023 20:15:36.102894 (+ 7213us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.103544 (+ 650us) write_op.cc:301] APPLY: starting
1023 20:15:36.103554 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.103585 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.103614 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.103657 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.103683 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:36.103691 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:36.103752 (+ 61us) log.cc:844] Serialized 23 byte log entry
1023 20:15:36.103834 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.103849 (+ 15us) write_op.cc:454] Released schema lock
1023 20:15:36.103855 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":88208128,"child_traces":[["op",{"apply.queue_time_us":620,"apply.run_cpu_time_us":372,"apply.run_wall_time_us":372,"num_ops":1,"prepare.queue_time_us":47827,"prepare.run_cpu_time_us":526,"prepare.run_wall_time_us":525,"raft.queue_time_us":429,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":7332,"thread_start_us":24221,"threads_started":1}]]}
I20251023 20:15:36.335753 23217 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 79) took 7749 ms. Trace:
I20251023 20:15:36.335862 23217 rpcz_store.cc:276] 1023 20:15:28.586448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.537619 (+6951171us) service_pool.cc:225] Handling call
1023 20:15:35.538071 (+ 452us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:35.565178 (+ 27107us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:35.565229 (+ 51us) catalog_manager.cc:2824] Locking tablets
1023 20:15:35.565354 (+ 125us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:35.604949 (+ 39595us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.604959 (+ 10us) write_op.cc:454] Released schema lock
1023 20:15:35.605069 (+ 110us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:35.605184 (+ 115us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:36.322474 (+717290us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:36.335506 (+ 13032us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.566327 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.566479 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.566487 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:35.566490 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:35.566687 (+ 197us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.566704 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.566709 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:35.566957 (+ 248us) tablet.cc:699] Row locks acquired
1023 20:15:35.566960 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:35.567003 (+ 43us) write_op.cc:270] Start()
1023 20:15:35.567043 (+ 40us) write_op.cc:276] Timestamp: P: 1761250535566950 usec, L: 0
1023 20:15:35.567047 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.567232 (+ 185us) log.cc:844] Serialized 734 byte log entry
1023 20:15:35.573392 (+ 6160us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.573640 (+ 248us) write_op.cc:301] APPLY: starting
1023 20:15:35.573657 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.573713 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.573746 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.573828 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.573864 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:35.573872 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:35.573957 (+ 85us) log.cc:844] Serialized 29 byte log entry
1023 20:15:35.574025 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.574048 (+ 23us) write_op.cc:454] Released schema lock
1023 20:15:35.574055 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":180,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":484,"num_ops":2,"prepare.queue_time_us":185,"prepare.run_cpu_time_us":1021,"prepare.run_wall_time_us":1021,"raft.queue_time_us":3611,"raft.run_cpu_time_us":629,"raft.run_wall_time_us":628,"replication_time_us":6313}]]}
I20251023 20:15:36.336580 23217 master_service.cc:954] ReplaceTablet: received request to replace tablet 3befdfc85bca42bb816e11d1dd832c8c from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:36.337391 23216 master_service.cc:954] ReplaceTablet: received request to replace tablet 60e8942deb7544a09b21c3d1c2a3f99f from {username='slave'} at 127.0.0.1:55414
I20251023 20:15:36.337729 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet a1efa4fcaeb049b484ca59b12025ce9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:35 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.338971 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet a1efa4fcaeb049b484ca59b12025ce9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:35 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.341193 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet 857da3642f0846ce92aaed0491e89644 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.341527 23787 tablet_bootstrap.cc:654] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.341889 23635 tablet_service.cc:1552] Processing DeleteTablet for tablet a841ffdc6cdd432dac7afe2e2b53f224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:21 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.342362 23637 tablet_service.cc:1552] Processing DeleteTablet for tablet 39a62c97e0a3459e91c24c1c2f92389a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:24 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.342938 23646 tablet_service.cc:1552] Processing DeleteTablet for tablet 370a790a053446b2b06314c7db242270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.343519 23648 tablet_service.cc:1552] Processing DeleteTablet for tablet c1f9935f5e4f4daa9df8fe1611ed76c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.344128 23644 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e19859a65a144598e1612be89daa3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:25 UTC) from {username='slave'} at 127.0.0.1:60202
W20251023 20:15:36.346747 23218 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.131:53879 (request call id 197) took 2159 ms (client timeout 1000 ms). Trace:
I20251023 20:15:36.347221 23790 tablet_bootstrap.cc:654] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.350162 25262 consensus_queue.cc:1048] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.353902 23787 tablet_bootstrap.cc:492] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.354071 23787 ts_tablet_manager.cc:1403] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20251023 20:15:36.354825 23787 raft_consensus.cc:359] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.355018 23787 raft_consensus.cc:385] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.355100 23787 raft_consensus.cc:740] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.355384 23787 consensus_queue.cc:260] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.355965 23787 ts_tablet_manager.cc:1434] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:36.356230 23787 tablet_bootstrap.cc:492] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.358407 23787 tablet_bootstrap.cc:654] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:36.346758 23212 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.127.129:42955 (request call id 196) took 5034 ms (client timeout 1000 ms). Trace:
W20251023 20:15:36.363994 23212 rpcz_store.cc:269] 1023 20:15:31.312833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.408650 (+ 95817us) service_pool.cc:225] Handling call
1023 20:15:36.346720 (+4938070us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:33.445587 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:33.445747 (+ 160us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:33.445755 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:33.445757 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:33.446003 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:33.446016 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:33.446021 (+ 5us) tablet.cc:683] Acquiring locks for 5 operations
1023 20:15:33.446204 (+ 183us) tablet.cc:699] Row locks acquired
1023 20:15:33.446207 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:33.446252 (+ 45us) write_op.cc:270] Start()
1023 20:15:33.446289 (+ 37us) write_op.cc:276] Timestamp: P: 1761250533444799 usec, L: 0
1023 20:15:33.446293 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:33.446516 (+ 223us) log.cc:844] Serialized 2069 byte log entry
1023 20:15:33.565114 (+118598us) op_driver.cc:464] REPLICATION: finished
1023 20:15:33.616325 (+ 51211us) write_op.cc:301] APPLY: starting
1023 20:15:33.616342 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:33.616420 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:33.616473 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:33.616612 (+ 139us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:33.616653 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
1023 20:15:33.616661 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:33.616813 (+ 152us) log.cc:844] Serialized 47 byte log entry
1023 20:15:33.616903 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:33.616927 (+ 24us) write_op.cc:454] Released schema lock
1023 20:15:33.616935 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":90472,"reported_tablets":235,"child_traces":[["op",{"apply.queue_time_us":51113,"apply.run_cpu_time_us":712,"apply.run_wall_time_us":712,"num_ops":5,"prepare.queue_time_us":222473,"prepare.run_cpu_time_us":1249,"prepare.run_wall_time_us":20597,"raft.queue_time_us":48,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":280,"replication_time_us":118751,"thread_start_us":293644,"threads_started":3,"wal-append.queue_time_us":12774}]]}
I20251023 20:15:36.364524 25172 raft_consensus.cc:515] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:36.365247 25470 consensus_meta.cc:287] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.877s user 0.003s sys 0.000s
I20251023 20:15:36.365355 25470 raft_consensus.cc:515] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.365955 25470 leader_election.cc:290] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:36.365788 23218 rpcz_store.cc:269] 1023 20:15:34.187593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:34.947258 (+759665us) service_pool.cc:225] Handling call
1023 20:15:36.346686 (+1399428us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:35.520045 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:35.520180 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:35.520188 (+ 8us) write_op.cc:435] Acquired schema lock
1023 20:15:35.520191 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:35.520312 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:35.520327 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:35.520331 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:35.520409 (+ 78us) tablet.cc:699] Row locks acquired
1023 20:15:35.520412 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:35.520453 (+ 41us) write_op.cc:270] Start()
1023 20:15:35.520493 (+ 40us) write_op.cc:276] Timestamp: P: 1761250535520405 usec, L: 0
1023 20:15:35.520497 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:35.520703 (+ 206us) log.cc:844] Serialized 356 byte log entry
1023 20:15:35.525866 (+ 5163us) op_driver.cc:464] REPLICATION: finished
1023 20:15:35.525993 (+ 127us) write_op.cc:301] APPLY: starting
1023 20:15:35.526011 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:35.526055 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:35.526079 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:35.526135 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:35.526169 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:35.526176 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:35.526261 (+ 85us) log.cc:844] Serialized 23 byte log entry
1023 20:15:35.526324 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.526344 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:35.526351 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:35.526424 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:35.526428 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":76276,"reported_tablets":197,"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":433,"apply.run_wall_time_us":432,"num_ops":1,"prepare.queue_time_us":1088,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":779,"raft.queue_time_us":407,"raft.run_cpu_time_us":667,"raft.run_wall_time_us":920,"replication_time_us":5312,"spinlock_wait_cycles":510336}]]}
I20251023 20:15:36.366266 23665 raft_consensus.cc:2468] T 53f33020319d48c791e1156c67b55be6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.348839 23210 rpcz_store.cc:276] 1023 20:15:31.503149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.277104 (+4773955us) service_pool.cc:225] Handling call
1023 20:15:36.285841 (+ 8737us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a06027a3 00007f8997106de6 00007f8997106b15 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:36.287632 (+ 1791us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:36.288806 (+ 1174us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:36.309244 (+ 20438us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:36.309341 (+ 97us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:36.309436 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.289326 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.289427 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.289433 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:36.289435 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.289522 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.289544 (+ 22us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.289548 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.289617 (+ 69us) tablet.cc:699] Row locks acquired
1023 20:15:36.289620 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.289652 (+ 32us) write_op.cc:270] Start()
1023 20:15:36.289683 (+ 31us) write_op.cc:276] Timestamp: P: 1761250536289633 usec, L: 0
1023 20:15:36.289686 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.289859 (+ 173us) log.cc:844] Serialized 433 byte log entry
1023 20:15:36.293903 (+ 4044us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.294079 (+ 176us) write_op.cc:301] APPLY: starting
1023 20:15:36.294098 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.294152 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.296364 (+ 2212us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.296499 (+ 135us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.296549 (+ 50us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:36.296558 (+ 9us) write_op.cc:312] APPLY: finished
1023 20:15:36.296655 (+ 97us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.296728 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.296752 (+ 24us) write_op.cc:454] Released schema lock
1023 20:15:36.296761 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:36.304076 (+ 7315us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.304083 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":19497216,"child_traces":[["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":595,"apply.run_wall_time_us":2746,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":686,"prepare.run_wall_time_us":14737,"raft.queue_time_us":228,"raft.run_cpu_time_us":558,"raft.run_wall_time_us":557,"replication_time_us":4187}]]}
I20251023 20:15:36.355821 23215 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 94) took 4852 ms. Trace:
I20251023 20:15:36.368109 23215 rpcz_store.cc:276] 1023 20:15:31.503696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.306105 (+4802409us) service_pool.cc:225] Handling call
1023 20:15:36.306986 (+ 881us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:36.307435 (+ 449us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:36.355565 (+ 48130us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:36.355677 (+ 112us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:36.355783 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.308229 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.308336 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.308341 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.308344 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:36.308432 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.308445 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.308451 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.308525 (+ 74us) tablet.cc:699] Row locks acquired
1023 20:15:36.308527 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:36.308585 (+ 58us) write_op.cc:270] Start()
1023 20:15:36.308619 (+ 34us) write_op.cc:276] Timestamp: P: 1761250536308565 usec, L: 0
1023 20:15:36.308623 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.308796 (+ 173us) log.cc:844] Serialized 433 byte log entry
1023 20:15:36.352698 (+ 43902us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.352861 (+ 163us) write_op.cc:301] APPLY: starting
1023 20:15:36.352881 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.352932 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.352966 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.353120 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.353157 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:36.353164 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.353249 (+ 85us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.353312 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.353334 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:36.353341 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1023 20:15:36.362116 (+ 8775us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.362123 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":547,"apply.run_wall_time_us":546,"num_ops":2,"prepare.queue_time_us":277,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":53877,"raft.queue_time_us":28607,"raft.run_cpu_time_us":643,"raft.run_wall_time_us":643,"replication_time_us":44043}]]}
I20251023 20:15:36.370574 23212 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_b98d2ec04db54e72a067731a38d21a56" } new_table_name: "default.Table_12e6cb242b54467f9041afc60fa47eb3" modify_external_catalogs: true
I20251023 20:15:36.372020 23215 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_94f35966189741de98430b366fccb99f" } modify_external_catalogs: true
I20251023 20:15:36.372191 23215 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_94f35966189741de98430b366fccb99f" } modify_external_catalogs: true
I20251023 20:15:36.378753 25172 leader_election.cc:290] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.379447 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b] alter complete (version 1)
I20251023 20:15:36.387109 23787 tablet_bootstrap.cc:492] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.387144 23210 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_5ba404e4eee84ed4bf30bf1be1236aa5" } modify_external_catalogs: true
I20251023 20:15:36.387432 23787 ts_tablet_manager.cc:1403] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.006s
I20251023 20:15:36.387666 23218 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_c07286b67c934b439529460275aa2631" } modify_external_catalogs: true
I20251023 20:15:36.388434 23787 raft_consensus.cc:359] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.388649 23787 raft_consensus.cc:385] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.388722 23787 raft_consensus.cc:740] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.388955 23787 consensus_queue.cc:260] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.389619 23787 ts_tablet_manager.cc:1434] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:36.389912 23218 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_c07286b67c934b439529460275aa2631" } modify_external_catalogs: true
I20251023 20:15:36.390076 23787 tablet_bootstrap.cc:492] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.390339 23210 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_5ba404e4eee84ed4bf30bf1be1236aa5" } modify_external_catalogs: true
I20251023 20:15:36.391947 23787 tablet_bootstrap.cc:654] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.392676 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "473324b22da14618a84aa1136d081993" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.392846 23665 raft_consensus.cc:3060] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.393724 23470 leader_election.cc:304] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.397006 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "473324b22da14618a84aa1136d081993" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:36.397230 23389 raft_consensus.cc:3060] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.397818 23214 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55414 (request call id 95) took 4890 ms. Trace:
I20251023 20:15:36.397961 23214 rpcz_store.cc:276] 1023 20:15:31.507116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.324231 (+4817115us) service_pool.cc:225] Handling call
1023 20:15:36.325249 (+ 1018us) catalog_manager.cc:2101] Acquired catalog manager lock
1023 20:15:36.325687 (+ 438us) catalog_manager.cc:2147] Created new table and tablet info
1023 20:15:36.397561 (+ 71874us) catalog_manager.cc:2207] Wrote table and tablets to system table
1023 20:15:36.397666 (+ 105us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1023 20:15:36.397759 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.387992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.388597 (+ 605us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.388603 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:36.388605 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.388836 (+ 231us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.388849 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.388853 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.388933 (+ 80us) tablet.cc:699] Row locks acquired
1023 20:15:36.388935 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:36.388971 (+ 36us) write_op.cc:270] Start()
1023 20:15:36.389005 (+ 34us) write_op.cc:276] Timestamp: P: 1761250536388945 usec, L: 0
1023 20:15:36.389008 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.389213 (+ 205us) log.cc:844] Serialized 433 byte log entry
1023 20:15:36.396498 (+ 7285us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.396666 (+ 168us) write_op.cc:301] APPLY: starting
1023 20:15:36.396682 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.396728 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.396765 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.396884 (+ 119us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.396920 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1023 20:15:36.396927 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.397007 (+ 80us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.397097 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.397119 (+ 22us) write_op.cc:454] Released schema lock
1023 20:15:36.397126 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":119,"apply.run_cpu_time_us":528,"apply.run_wall_time_us":527,"num_ops":2,"prepare.queue_time_us":61832,"raft.queue_time_us":171,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":355,"replication_time_us":7459}]]}
I20251023 20:15:36.403286 25470 raft_consensus.cc:2804] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.403427 25470 raft_consensus.cc:697] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:36.403677 25470 consensus_queue.cc:237] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.405808 23214 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_331b508de7624556842102252e4ccc38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.407130 23214 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_331b508de7624556842102252e4ccc38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.409741 23787 tablet_bootstrap.cc:492] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.409902 23787 ts_tablet_manager.cc:1403] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.003s
I20251023 20:15:36.410741 23787 raft_consensus.cc:359] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.413722 23787 raft_consensus.cc:385] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.413806 23787 raft_consensus.cc:740] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.414028 23787 consensus_queue.cc:260] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.414625 23787 ts_tablet_manager.cc:1434] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:36.414903 23787 tablet_bootstrap.cc:492] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.416929 23787 tablet_bootstrap.cc:654] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.423604 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet a1efa4fcaeb049b484ca59b12025ce9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:35 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.423909 23217 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 98) took 4281 ms. Trace:
I20251023 20:15:36.424047 23217 rpcz_store.cc:276] 1023 20:15:32.142364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.336488 (+4194124us) service_pool.cc:225] Handling call
1023 20:15:36.423869 (+ 87381us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:36.424322 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 419e41d7a37f4b888cb242b089c28790 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:22 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.425660 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26a680ad3aed487593bd1b63004dbbfc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:36.425848 23530 raft_consensus.cc:3060] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.430127 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 21a508e875c449c5831968562263f1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:35 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.434697 25619 consensus_queue.cc:1048] T 8e718cb7205643fd8c3045c3fab30d3e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.437166 25359 raft_consensus.cc:493] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.437299 25359 raft_consensus.cc:515] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.437880 25359 leader_election.cc:290] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.438439 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.439208 23605 leader_election.cc:343] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.440807 23217 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_e177ffb3adcc48dc84739ca2bdc1ae6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.441423 23217 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e177ffb3adcc48dc84739ca2bdc1ae6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.441861 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:36.442265 23604 leader_election.cc:343] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.442370 23604 leader_election.cc:304] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.442713 25359 raft_consensus.cc:2749] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:36.445989 23333 leader_election.cc:343] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 26a680ad3aed487593bd1b63004dbbfc
W20251023 20:15:36.446588 23333 consensus_peers.cc:597] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dafb9db710f94c328186e3179bf0b319. This is attempt 6: this message will repeat every 5th retry.
I20251023 20:15:36.450495 23212 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:55414 (request call id 99) took 3025 ms. Trace:
I20251023 20:15:36.450608 23212 rpcz_store.cc:276] 1023 20:15:33.425287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.364617 (+2939330us) service_pool.cc:225] Handling call
1023 20:15:36.370512 (+ 5895us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0604a36 00007f899710a766 00007f899710a495 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:36.370760 (+ 248us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:36.370959 (+ 199us) catalog_manager.cc:3708] Apply alter schema
1023 20:15:36.371159 (+ 200us) catalog_manager.cc:3740] Acquired catalog manager lock
1023 20:15:36.371231 (+ 72us) catalog_manager.cc:3898] Updating metadata on disk
1023 20:15:36.450012 (+ 78781us) catalog_manager.cc:3929] Committing alterations to in-memory state
1023 20:15:36.450437 (+ 425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.442649 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.442741 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.442746 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.442748 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.442867 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.442912 (+ 45us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.442917 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1023 20:15:36.442970 (+ 53us) tablet.cc:699] Row locks acquired
1023 20:15:36.442972 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:36.443001 (+ 29us) write_op.cc:270] Start()
1023 20:15:36.443029 (+ 28us) write_op.cc:276] Timestamp: P: 1761250536442973 usec, L: 0
1023 20:15:36.443033 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.443183 (+ 150us) log.cc:844] Serialized 405 byte log entry
1023 20:15:36.449419 (+ 6236us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.449586 (+ 167us) write_op.cc:301] APPLY: starting
1023 20:15:36.449602 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.449643 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.449665 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.449711 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.449740 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:36.449747 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.449819 (+ 72us) log.cc:844] Serialized 23 byte log entry
1023 20:15:36.449877 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.449894 (+ 17us) write_op.cc:454] Released schema lock
1023 20:15:36.449900 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":12967808,"child_traces":[["op",{"apply.queue_time_us":119,"apply.run_cpu_time_us":372,"apply.run_wall_time_us":371,"num_ops":1,"prepare.queue_time_us":70948,"raft.queue_time_us":34,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":279,"replication_time_us":6360}]]}
I20251023 20:15:36.457409 23786 tablet_bootstrap.cc:492] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.457544 23786 ts_tablet_manager.cc:1403] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.137s user 0.016s sys 0.041s
I20251023 20:15:36.458201 23786 raft_consensus.cc:359] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.458359 23786 raft_consensus.cc:385] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.458400 23786 raft_consensus.cc:740] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.458566 23786 consensus_queue.cc:260] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.458994 23786 ts_tablet_manager.cc:1434] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:36.459268 23786 tablet_bootstrap.cc:492] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.461149 23786 tablet_bootstrap.cc:654] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.463099 23212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_8d1521c5f05d4bb5a060ed68c5fa6ca5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.463671 23212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d1521c5f05d4bb5a060ed68c5fa6ca5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.464401 23790 tablet_bootstrap.cc:492] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.464416 23398 raft_consensus.cc:2468] T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.466298 23790 ts_tablet_manager.cc:1403] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.179s user 0.004s sys 0.000s
I20251023 20:15:36.467171 23790 raft_consensus.cc:359] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.468138 25514 catalog_manager.cc:1269] Loaded cluster ID: 67ec0cebd86e4322972b698542d64fab
I20251023 20:15:36.468444 25514 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: loading cluster ID for follower catalog manager: success
I20251023 20:15:36.468634 23790 raft_consensus.cc:385] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:36.469972 23532 consensus_meta.cc:287] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.933s user 0.002s sys 0.000s
I20251023 20:15:36.470058 23532 raft_consensus.cc:2468] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:36.470384 23666 raft_consensus.cc:2468] T 82ae1a94b011412c851cd96946744fc4 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:36.470901 25282 ts_tablet_manager.cc:1929] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:36.471001 25282 log.cc:1199] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/1822fdf1f5f44dd4a742aa3c2f8443cf
I20251023 20:15:36.471284 25282 ts_tablet_manager.cc:1950] T 1822fdf1f5f44dd4a742aa3c2f8443cf P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:36.471714 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 400) took 1170 ms. Trace:
I20251023 20:15:36.471827 25492 rpcz_store.cc:276] 1023 20:15:35.301287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.301396 (+ 109us) service_pool.cc:225] Handling call
1023 20:15:35.338484 (+ 37088us) ts_tablet_manager.cc:1151] Deleting tablet 514cb3b1ed024041ac160a7d9acff618
1023 20:15:35.338491 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:35.522792 (+184301us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:35.586087 (+ 63295us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.471695 (+885608us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":36886}
I20251023 20:15:36.472254 25492 tablet_replica.cc:333] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:36.473389 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet 514cb3b1ed024041ac160a7d9acff618 (table default.Table_7a413e9be0f843bc97a50978fefdec72 [id=03e212979a484619a37f1d1e4e27c50a]) successfully deleted
I20251023 20:15:36.474325 23219 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573] alter complete (version 1)
I20251023 20:15:36.474972 25492 raft_consensus.cc:2243] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:36.475134 25492 raft_consensus.cc:2272] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:36.475646 25492 ts_tablet_manager.cc:1916] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:36.475122 25172 raft_consensus.cc:493] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.479029 25172 raft_consensus.cc:515] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.479629 25172 leader_election.cc:290] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.468880 23790 raft_consensus.cc:740] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.480067 23790 consensus_queue.cc:260] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.481105 23790 ts_tablet_manager.cc:1434] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251023 20:15:36.486119 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d22aa40e5844d78a8679a3ad1b08da9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.486318 23532 raft_consensus.cc:2468] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:36.487155 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d22aa40e5844d78a8679a3ad1b08da9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.487331 23666 raft_consensus.cc:2468] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:36.489478 23333 leader_election.cc:304] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:36.489734 25172 raft_consensus.cc:2804] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.489862 25172 raft_consensus.cc:697] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:36.490090 25172 consensus_queue.cc:237] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.494386 25276 raft_consensus.cc:493] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.494536 25276 raft_consensus.cc:515] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.495188 25276 leader_election.cc:290] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.496068 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.496639 23605 leader_election.cc:343] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.497622 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:36.498114 23604 leader_election.cc:343] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.498225 23604 leader_election.cc:304] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.498621 25276 raft_consensus.cc:2749] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.502018 23333 leader_election.cc:304] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.502252 25172 raft_consensus.cc:2804] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.502382 25172 raft_consensus.cc:493] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.502493 25172 raft_consensus.cc:3060] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.504949 23333 leader_election.cc:304] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:36.505221 25235 raft_consensus.cc:2804] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.505338 25235 raft_consensus.cc:697] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:36.505607 25235 consensus_queue.cc:237] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.506649 23786 tablet_bootstrap.cc:492] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.506821 23786 ts_tablet_manager.cc:1403] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.006s
I20251023 20:15:36.507462 23786 raft_consensus.cc:359] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.507669 23786 raft_consensus.cc:385] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.507750 23786 raft_consensus.cc:740] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.507988 23786 consensus_queue.cc:260] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.508289 23214 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_58a322a6e70c4cd58aa1cca39508e78d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:36.508481 23786 ts_tablet_manager.cc:1434] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.508746 23786 tablet_bootstrap.cc:492] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:36.509027 23214 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58a322a6e70c4cd58aa1cca39508e78d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.510536 23786 tablet_bootstrap.cc:654] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.512194 25644 tablet.cc:1722] T 244fa327fad74aad9e57a7fbcbf908bb P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:36.512588 25646 tablet.cc:1722] T 244fa327fad74aad9e57a7fbcbf908bb P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:36.519055 23790 tablet_bootstrap.cc:492] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.520630 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 170) took 7742 ms. Trace:
I20251023 20:15:36.520740 24780 rpcz_store.cc:276] 1023 20:15:28.778413 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.778451 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:35.941155 (+7162704us) ts_tablet_manager.cc:1151] Deleting tablet 951b2276d3324c2d999c8d7fb97767bc
1023 20:15:35.941166 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.168358 (+227192us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.291267 (+122909us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.520606 (+229339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7162565}
I20251023 20:15:36.521121 24780 tablet_replica.cc:333] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:36.521340 24780 raft_consensus.cc:2243] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:36.521696 24780 raft_consensus.cc:2272] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:36.522110 23216 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 60e8942deb7544a09b21c3d1c2a3f99f
I20251023 20:15:36.522260 24780 ts_tablet_manager.cc:1916] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:36.522641 23216 catalog_manager.cc:6719] ReplaceTablet: tablet 60e8942deb7544a09b21c3d1c2a3f99f deleted and replaced by tablet 94bb861a96cd40748d35e909418b813a
I20251023 20:15:36.522755 23216 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 97) took 4799 ms. Trace:
I20251023 20:15:36.525908 23790 tablet_bootstrap.cc:654] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.526057 23216 rpcz_store.cc:276] 1023 20:15:31.723349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.335644 (+4612295us) service_pool.cc:225] Handling call
1023 20:15:36.522726 (+187082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.510234 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.510333 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.510338 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.510340 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.510436 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.510447 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.510451 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.510547 (+ 96us) tablet.cc:699] Row locks acquired
1023 20:15:36.510549 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:36.510579 (+ 30us) write_op.cc:270] Start()
1023 20:15:36.510607 (+ 28us) write_op.cc:276] Timestamp: P: 1761250536510546 usec, L: 0
1023 20:15:36.510610 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.510762 (+ 152us) log.cc:844] Serialized 631 byte log entry
1023 20:15:36.520977 (+ 10215us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.521159 (+ 182us) write_op.cc:301] APPLY: starting
1023 20:15:36.521177 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.521224 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.521255 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.521364 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.521427 (+ 63us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:36.521433 (+ 6us) write_op.cc:312] APPLY: finished
1023 20:15:36.521519 (+ 86us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.521594 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.521615 (+ 21us) write_op.cc:454] Released schema lock
1023 20:15:36.521622 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":530,"apply.run_wall_time_us":530,"num_ops":2,"prepare.queue_time_us":82463,"raft.queue_time_us":44,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":339,"replication_time_us":10333}]]}
I20251023 20:15:36.529735 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 60e8942deb7544a09b21c3d1c2a3f99f with delete_type TABLET_DATA_DELETED (replaced by tablet 94bb861a96cd40748d35e909418b813a) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.530274 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 60e8942deb7544a09b21c3d1c2a3f99f with delete_type TABLET_DATA_DELETED (replaced by tablet 94bb861a96cd40748d35e909418b813a) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.540401 25594 raft_consensus.cc:493] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.540580 25594 raft_consensus.cc:515] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.541199 25594 leader_election.cc:290] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.544477 25600 raft_consensus.cc:493] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.545210 25600 raft_consensus.cc:515] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.546980 23216 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_0c73476a1a6d41eeb3ed876c729c6d21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.547678 23216 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c73476a1a6d41eeb3ed876c729c6d21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.548561 25600 leader_election.cc:290] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.554607 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 951b2276d3324c2d999c8d7fb97767bc (table default.Table_441bf8050be9453c85aef2cb7ac63ea4 [id=425e2178647a42aaa99cef01f06523f1]) successfully deleted
I20251023 20:15:36.555415 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e2fec8f4e14036af714427481f2f36" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.555621 23669 raft_consensus.cc:2468] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:36.556258 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e2fec8f4e14036af714427481f2f36" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.556465 23536 raft_consensus.cc:2468] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:36.558890 23213 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 98a6a3425a49403e99c8ade114166e46
I20251023 20:15:36.559085 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.559243 23669 raft_consensus.cc:2468] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:36.559324 23213 catalog_manager.cc:6719] ReplaceTablet: tablet 98a6a3425a49403e99c8ade114166e46 deleted and replaced by tablet c6577d1bf92444f9a9ea1b5f5e7f005f
I20251023 20:15:36.559423 23213 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:55414 (request call id 96) took 5025 ms. Trace:
I20251023 20:15:36.559559 23470 leader_election.cc:304] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.559512 23213 rpcz_store.cc:276] 1023 20:15:31.534243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.334084 (+4799841us) service_pool.cc:225] Handling call
1023 20:15:36.559397 (+225313us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.549274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.549395 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.549400 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.549403 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:36.549515 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.549529 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.549545 (+ 16us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.549623 (+ 78us) tablet.cc:699] Row locks acquired
1023 20:15:36.549626 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.549657 (+ 31us) write_op.cc:270] Start()
1023 20:15:36.549690 (+ 33us) write_op.cc:276] Timestamp: P: 1761250536549621 usec, L: 0
1023 20:15:36.549694 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.549862 (+ 168us) log.cc:844] Serialized 631 byte log entry
1023 20:15:36.557914 (+ 8052us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.558067 (+ 153us) write_op.cc:301] APPLY: starting
1023 20:15:36.558084 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.558135 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.558177 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.558308 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.558383 (+ 75us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1023 20:15:36.558391 (+ 8us) write_op.cc:312] APPLY: finished
1023 20:15:36.558469 (+ 78us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.558546 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.558570 (+ 24us) write_op.cc:454] Released schema lock
1023 20:15:36.558578 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":579,"apply.run_wall_time_us":578,"num_ops":2,"prepare.queue_time_us":120410,"raft.queue_time_us":273,"raft.run_cpu_time_us":588,"raft.run_wall_time_us":586,"replication_time_us":8187}]]}
I20251023 20:15:36.559903 25600 raft_consensus.cc:2804] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.559971 25600 raft_consensus.cc:493] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.560026 25600 raft_consensus.cc:3060] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.563287 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 98a6a3425a49403e99c8ade114166e46 with delete_type TABLET_DATA_DELETED (replaced by tablet c6577d1bf92444f9a9ea1b5f5e7f005f) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.564460 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 371) took 3037 ms. Trace:
I20251023 20:15:36.564553 25282 rpcz_store.cc:276] 1023 20:15:33.527189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.527250 (+ 61us) service_pool.cc:225] Handling call
1023 20:15:36.260686 (+2733436us) ts_tablet_manager.cc:1151] Deleting tablet 1822fdf1f5f44dd4a742aa3c2f8443cf
1023 20:15:36.260696 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.366588 (+105892us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.470780 (+104192us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.564437 (+ 93657us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2729037}
I20251023 20:15:36.564898 25282 tablet_replica.cc:333] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:36.564958 25454 consensus_meta.cc:287] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.464s user 0.004s sys 0.000s
I20251023 20:15:36.565133 25282 raft_consensus.cc:2243] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:36.565060 25454 raft_consensus.cc:515] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.565284 25282 raft_consensus.cc:2272] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:36.565690 25454 leader_election.cc:290] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.565805 25282 ts_tablet_manager.cc:1916] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:36.565966 24297 consensus_meta.cc:287] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.570s user 0.002s sys 0.001s
I20251023 20:15:36.566066 24297 raft_consensus.cc:515] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.566632 24297 leader_election.cc:290] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.569036 23213 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_6de6ff0130d74ad894cfa487113e1b19" } new_table_name: "default.Table_f021ca39359245598311be2bc68bbbcc" modify_external_catalogs: true
I20251023 20:15:36.575889 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e658aa360e524a9f9bdb35c1f68ae426" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.576071 23669 raft_consensus.cc:3060] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.577646 23665 raft_consensus.cc:2468] T 473324b22da14618a84aa1136d081993 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.578135 23470 leader_election.cc:304] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.578219 23215 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet bc7793d1370d47ac8ee66d0246aba8fe
I20251023 20:15:36.578403 25454 raft_consensus.cc:2804] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.578507 25454 raft_consensus.cc:697] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:36.578773 23215 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 100) took 3149 ms. Trace:
I20251023 20:15:36.578751 25454 consensus_queue.cc:237] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.578858 23215 rpcz_store.cc:276] 1023 20:15:33.429167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.368677 (+2939510us) service_pool.cc:225] Handling call
1023 20:15:36.371970 (+ 3293us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0603796 00007f8997108aa6 00007f89971087d5 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:36.372327 (+ 357us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:36.372449 (+ 122us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:36.372501 (+ 52us) catalog_manager.cc:2824] Locking tablets
1023 20:15:36.429251 (+ 56750us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:36.578049 (+148798us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:36.578134 (+ 85us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:36.578170 (+ 36us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:36.578723 (+ 553us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.572113 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.572220 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.572225 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.572227 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.572336 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.572348 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.572354 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.572448 (+ 94us) tablet.cc:699] Row locks acquired
1023 20:15:36.572451 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.572485 (+ 34us) write_op.cc:270] Start()
1023 20:15:36.572517 (+ 32us) write_op.cc:276] Timestamp: P: 1761250536572446 usec, L: 0
1023 20:15:36.572520 (+ 3us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.572709 (+ 189us) log.cc:844] Serialized 734 byte log entry
1023 20:15:36.577391 (+ 4682us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.577525 (+ 134us) write_op.cc:301] APPLY: starting
1023 20:15:36.577552 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.577596 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.577627 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.577694 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.577727 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:36.577734 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.577816 (+ 82us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.577879 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.577899 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:36.577924 (+ 25us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":7054592,"child_traces":[["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":460,"apply.run_wall_time_us":459,"num_ops":2,"prepare.queue_time_us":142298,"raft.queue_time_us":68,"raft.run_cpu_time_us":505,"raft.run_wall_time_us":506,"replication_time_us":4844}]]}
I20251023 20:15:36.582234 25514 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: acquiring CA information for follower catalog manager: success
I20251023 20:15:36.587963 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet bc7793d1370d47ac8ee66d0246aba8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.588840 23786 tablet_bootstrap.cc:492] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.595299 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.595414 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e658aa360e524a9f9bdb35c1f68ae426" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:36.595597 23387 raft_consensus.cc:3060] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.596418 23375 tablet_service.cc:1552] Processing DeleteTablet for tablet 60e8942deb7544a09b21c3d1c2a3f99f with delete_type TABLET_DATA_DELETED (replaced by tablet 94bb861a96cd40748d35e909418b813a) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.596571 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet bc7793d1370d47ac8ee66d0246aba8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.596611 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 98a6a3425a49403e99c8ade114166e46 with delete_type TABLET_DATA_DELETED (replaced by tablet c6577d1bf92444f9a9ea1b5f5e7f005f) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.596789 23786 ts_tablet_manager.cc:1403] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.003s
I20251023 20:15:36.597788 23786 raft_consensus.cc:359] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.597945 25489 consensus_queue.cc:1048] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.599618 23786 raft_consensus.cc:385] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.612473 23786 raft_consensus.cc:740] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.613113 23786 consensus_queue.cc:260] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.613268 23333 leader_election.cc:304] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:36.612783 23604 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 95006.8us
I20251023 20:15:36.613579 24297 raft_consensus.cc:2804] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.613663 24297 raft_consensus.cc:493] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.613720 24297 raft_consensus.cc:3060] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.614019 23786 ts_tablet_manager.cc:1434] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251023 20:15:36.614787 23786 tablet_bootstrap.cc:492] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.615185 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5161336598884742ad403fc88c9d9f0b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:36.615365 23532 raft_consensus.cc:3060] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.616420 25663 tablet.cc:1722] T 244fa327fad74aad9e57a7fbcbf908bb P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251023 20:15:36.616768 23786 tablet_bootstrap.cc:654] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.618189 24908 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" instance_seqno: 1761250513786292) as {username='slave'} at 127.22.127.130:55169; Asking this server to re-register.
I20251023 20:15:36.618594 23787 tablet_bootstrap.cc:492] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.618721 23787 ts_tablet_manager.cc:1403] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.204s user 0.002s sys 0.010s
I20251023 20:15:36.619033 23215 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_e6fec6ab98474529a2fdf84e3d3cdd2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:36.619458 23787 raft_consensus.cc:359] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.619604 23787 raft_consensus.cc:385] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.619652 23787 raft_consensus.cc:740] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
W20251023 20:15:36.619655 23215 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6fec6ab98474529a2fdf84e3d3cdd2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.619885 23787 consensus_queue.cc:260] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.620357 23787 ts_tablet_manager.cc:1434] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.620751 23787 tablet_bootstrap.cc:492] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.621690 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5161336598884742ad403fc88c9d9f0b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.621908 23666 raft_consensus.cc:3060] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.625222 23787 tablet_bootstrap.cc:654] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.628894 25644 tablet.cc:1722] T 559a70cf17ad4145acdb3033e01ad217 P 9bc6522db11e4960a9aad84435371a32: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251023 20:15:36.626546 23471 leader_election.cc:343] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.633934 25489 raft_consensus.cc:493] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.634078 25489 raft_consensus.cc:515] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.634696 25489 leader_election.cc:290] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.637574 25489 consensus_queue.cc:1048] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.638190 23210 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 92c50c4845524163834861438291a3ec
I20251023 20:15:36.638162 25489 consensus_queue.cc:1048] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.639081 23210 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 101) took 3173 ms. Trace:
I20251023 20:15:36.639716 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 92c50c4845524163834861438291a3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.640010 23210 rpcz_store.cc:276] 1023 20:15:33.465811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.369841 (+2904030us) service_pool.cc:225] Handling call
1023 20:15:36.387089 (+ 17248us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0603796 00007f8997108aa6 00007f89971087d5 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:36.390455 (+ 3366us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:36.390594 (+ 139us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:36.390646 (+ 52us) catalog_manager.cc:2824] Locking tablets
1023 20:15:36.432523 (+ 41877us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:36.637931 (+205408us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:36.638021 (+ 90us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:36.638066 (+ 45us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:36.639021 (+ 955us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.621011 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.621144 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.621149 (+ 5us) write_op.cc:435] Acquired schema lock
1023 20:15:36.621152 (+ 3us) tablet.cc:660] Decoding operations
1023 20:15:36.621361 (+ 209us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.621375 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.621380 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.621485 (+ 105us) tablet.cc:699] Row locks acquired
1023 20:15:36.621487 (+ 2us) write_op.cc:260] PREPARE: finished
1023 20:15:36.621524 (+ 37us) write_op.cc:270] Start()
1023 20:15:36.621570 (+ 46us) write_op.cc:276] Timestamp: P: 1761250536621481 usec, L: 0
1023 20:15:36.621574 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.621795 (+ 221us) log.cc:844] Serialized 799 byte log entry
1023 20:15:36.635604 (+ 13809us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.635768 (+ 164us) write_op.cc:301] APPLY: starting
1023 20:15:36.635787 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.635837 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.635864 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.635941 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.635978 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:36.635985 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.636090 (+ 105us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.636165 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.636188 (+ 23us) write_op.cc:454] Released schema lock
1023 20:15:36.636196 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":39047552,"child_traces":[["op",{"apply.queue_time_us":105,"apply.run_cpu_time_us":500,"apply.run_wall_time_us":500,"num_ops":2,"prepare.queue_time_us":187910,"raft.queue_time_us":75,"raft.run_cpu_time_us":579,"raft.run_wall_time_us":580,"replication_time_us":13994}]]}
I20251023 20:15:36.641244 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a6a2d2b25824dbdafaf2a9cabb0f061" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.643253 25619 consensus_queue.cc:1048] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.647657 25276 raft_consensus.cc:493] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.647585 25656 tablet.cc:1722] T 559a70cf17ad4145acdb3033e01ad217 P d220aba7c9c0400398ffde917376ee48: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:36.647794 25276 raft_consensus.cc:515] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:36.649667 23470 consensus_peers.cc:597] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1f9935f5e4f4daa9df8fe1611ed76c9. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:36.649730 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_b029eee1964748f3bb848b0f6309a357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251023 20:15:36.650288 25276 leader_election.cc:290] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:36.650548 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b029eee1964748f3bb848b0f6309a357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.650852 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ee1974eb5e343b7953dadda7d5799b1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.651355 23605 leader_election.cc:343] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.651638 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ee1974eb5e343b7953dadda7d5799b1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:36.652046 23604 leader_election.cc:343] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.652154 23604 leader_election.cc:304] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.652603 25276 raft_consensus.cc:2749] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.654775 25262 consensus_queue.cc:1048] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.656731 25262 raft_consensus.cc:493] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.656872 25262 raft_consensus.cc:515] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.657402 25262 leader_election.cc:290] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.658006 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19da816a483a4d5b9dc768ef697d54ec" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.658174 23662 raft_consensus.cc:2468] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:36.658682 23470 leader_election.cc:304] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.658991 25262 raft_consensus.cc:2804] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.659055 25262 raft_consensus.cc:493] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.659106 25262 raft_consensus.cc:3060] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.659477 23669 raft_consensus.cc:2468] T e658aa360e524a9f9bdb35c1f68ae426 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.659603 25492 ts_tablet_manager.cc:1929] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:36.659696 25492 log.cc:1199] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/c9bfbf0acdc84f2f8fe553694ee9897c
I20251023 20:15:36.659933 23470 leader_election.cc:304] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.660032 25492 ts_tablet_manager.cc:1950] T c9bfbf0acdc84f2f8fe553694ee9897c P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:36.660964 23218 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 690ffde75cd745b6a87925967d54471a: Sending DeleteTablet for 3 replicas of tablet 124703edbf08408a9ca69732674c0ad3
I20251023 20:15:36.661463 23218 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:55414 (request call id 102) took 1510 ms. Trace:
I20251023 20:15:36.661582 23218 rpcz_store.cc:276] 1023 20:15:35.150804 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.370157 (+1219353us) service_pool.cc:225] Handling call
1023 20:15:36.387594 (+ 17437us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x616000041254. stack: 00007f898c90f3e6 00007f898c90f1a0 00007f898b8ed29d 0000000000983974 000000000098393d 00007f899bbf553d 00007f899bbf5633 00007f89987bc261 00007f89a0475f1d 00007f89a0603796 00007f8997108aa6 00007f89971087d5 00007f8993884f1e 00007f8993883dfe 00007f899388a035 00007f899388a95f
1023 20:15:36.390046 (+ 2452us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1023 20:15:36.390176 (+ 130us) catalog_manager.cc:2815] Modifying in-memory table state
1023 20:15:36.390243 (+ 67us) catalog_manager.cc:2824] Locking tablets
1023 20:15:36.433277 (+ 43034us) catalog_manager.cc:2838] Removing table and tablets from system table
1023 20:15:36.660766 (+227489us) catalog_manager.cc:2856] Removing table from by-name map
1023 20:15:36.660849 (+ 83us) catalog_manager.cc:2872] Committing in-memory state
1023 20:15:36.660894 (+ 45us) catalog_manager.cc:2876] Aborting table tasks
1023 20:15:36.661430 (+ 536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1023 20:15:36.651930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1023 20:15:36.653769 (+ 1839us) write_op.cc:432] Acquiring schema lock in shared mode
1023 20:15:36.653775 (+ 6us) write_op.cc:435] Acquired schema lock
1023 20:15:36.653777 (+ 2us) tablet.cc:660] Decoding operations
1023 20:15:36.653939 (+ 162us) write_op.cc:620] Acquiring the partition lock for write op
1023 20:15:36.653951 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1023 20:15:36.653956 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1023 20:15:36.654055 (+ 99us) tablet.cc:699] Row locks acquired
1023 20:15:36.654058 (+ 3us) write_op.cc:260] PREPARE: finished
1023 20:15:36.654096 (+ 38us) write_op.cc:270] Start()
1023 20:15:36.654130 (+ 34us) write_op.cc:276] Timestamp: P: 1761250536654050 usec, L: 0
1023 20:15:36.654134 (+ 4us) op_driver.cc:348] REPLICATION: starting
1023 20:15:36.654409 (+ 275us) log.cc:844] Serialized 799 byte log entry
1023 20:15:36.658836 (+ 4427us) op_driver.cc:464] REPLICATION: finished
1023 20:15:36.658995 (+ 159us) write_op.cc:301] APPLY: starting
1023 20:15:36.659012 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1023 20:15:36.659058 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1023 20:15:36.659090 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1023 20:15:36.659158 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
1023 20:15:36.659192 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1023 20:15:36.659199 (+ 7us) write_op.cc:312] APPLY: finished
1023 20:15:36.659273 (+ 74us) log.cc:844] Serialized 29 byte log entry
1023 20:15:36.659332 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1023 20:15:36.659352 (+ 20us) write_op.cc:454] Released schema lock
1023 20:15:36.659359 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":16073600,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":433,"apply.run_wall_time_us":433,"num_ops":2,"prepare.queue_time_us":218129,"raft.queue_time_us":64,"raft.run_cpu_time_us":498,"raft.run_wall_time_us":499,"replication_time_us":4673}]]}
I20251023 20:15:36.662672 23666 raft_consensus.cc:2468] T 5161336598884742ad403fc88c9d9f0b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:36.662842 23508 tablet_service.cc:1552] Processing DeleteTablet for tablet 124703edbf08408a9ca69732674c0ad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:48312
I20251023 20:15:36.664057 25454 raft_consensus.cc:2804] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.664168 25454 raft_consensus.cc:697] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:36.664387 25454 consensus_queue.cc:237] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:36.664848 25466 consensus_meta.cc:287] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.919s user 0.002s sys 0.000s
I20251023 20:15:36.664947 25466 raft_consensus.cc:515] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:36.665267 23392 consensus_meta.cc:287] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.572s user 0.001s sys 0.001s
I20251023 20:15:36.665371 23392 raft_consensus.cc:2468] T a744214a24cc4bf28654ffa417ce0d3d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.665892 24780 ts_tablet_manager.cc:1929] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:36.666008 24780 log.cc:1199] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/dafb9db710f94c328186e3179bf0b319
I20251023 20:15:36.666363 24780 ts_tablet_manager.cc:1950] T dafb9db710f94c328186e3179bf0b319 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:36.679984 23218 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_03ddad2769a4450b81d0d8028293a0b2" } new_table_name: "default.Table_869b3f73ab4446369b8ab49294b2d32d" modify_external_catalogs: true
I20251023 20:15:36.683346 23790 tablet_bootstrap.cc:492] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.683480 23790 ts_tablet_manager.cc:1403] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.165s user 0.004s sys 0.023s
I20251023 20:15:36.684211 23790 raft_consensus.cc:359] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.684365 23790 raft_consensus.cc:385] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.684412 23790 raft_consensus.cc:740] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.684616 23790 consensus_queue.cc:260] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.684701 25466 leader_election.cc:290] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.685127 23790 ts_tablet_manager.cc:1434] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:36.685380 23790 tablet_bootstrap.cc:492] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.685827 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9be2a866cf46cfbec8db23a6d3f572" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.686017 23670 raft_consensus.cc:3060] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.687072 23790 tablet_bootstrap.cc:654] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.689999 23583 heartbeater.cc:461] Registering TS with master...
I20251023 20:15:36.678620 25514 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251023 20:15:36.697460 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 1822fdf1f5f44dd4a742aa3c2f8443cf (table default.Table_8e6b9df594194069b682dc1ece624a9e [id=759c4ec2b0544418a144bb7d3c02b952]) successfully deleted
I20251023 20:15:36.699442 23217 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_d78ebb55a1754b5bb096fab938774ce1" } modify_external_catalogs: true
I20251023 20:15:36.700204 23217 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_d78ebb55a1754b5bb096fab938774ce1" } modify_external_catalogs: true
I20251023 20:15:36.710723 25276 raft_consensus.cc:493] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.710922 25276 raft_consensus.cc:515] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.711484 25276 leader_election.cc:290] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.712746 25276 raft_consensus.cc:493] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.712888 25276 raft_consensus.cc:515] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.713449 25276 leader_election.cc:290] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.713828 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757d41e027ba461ea051b38a2d685407" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.714825 25646 tablet.cc:1722] T 559a70cf17ad4145acdb3033e01ad217 P ea6853efbbcc484eb7529c7e9b37a47a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251023 20:15:36.715240 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453eea5cdd2e47f29577c61b40ee59c0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.719970 23366 tablet_service.cc:1552] Processing DeleteTablet for tablet 92c50c4845524163834861438291a3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:38604
I20251023 20:15:36.720023 23371 tablet_service.cc:1552] Processing DeleteTablet for tablet 124703edbf08408a9ca69732674c0ad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:38604
W20251023 20:15:36.720155 23333 consensus_peers.cc:597] T 7e36bb5e61d34b35a373bd6a51a7ac5e P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e36bb5e61d34b35a373bd6a51a7ac5e. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:36.721439 23605 leader_election.cc:343] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.722076 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453eea5cdd2e47f29577c61b40ee59c0" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:36.722544 23604 leader_election.cc:343] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:36.723497 23604 leader_election.cc:343] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.723590 23604 leader_election.cc:304] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.724038 25276 raft_consensus.cc:2749] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.729825 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757d41e027ba461ea051b38a2d685407" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.730482 23605 leader_election.cc:343] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.730602 23605 leader_election.cc:304] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.731030 25276 raft_consensus.cc:2749] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.733385 23333 leader_election.cc:304] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:36.734064 25489 raft_consensus.cc:2804] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.734305 25489 raft_consensus.cc:697] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:36.734606 25489 consensus_queue.cc:237] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.735358 25594 raft_consensus.cc:493] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.735488 25594 raft_consensus.cc:515] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.736245 25594 leader_election.cc:290] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:36.744900 23333 leader_election.cc:343] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 9a6a2d2b25824dbdafaf2a9cabb0f061
I20251023 20:15:36.753868 23526 raft_consensus.cc:1275] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Refusing update from remote peer d220aba7c9c0400398ffde917376ee48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251023 20:15:36.754381 23526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3150a8478c144d285e999d5c9270fe0" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.759373 23670 raft_consensus.cc:2468] T ea9be2a866cf46cfbec8db23a6d3f572 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.759148 23790 tablet_bootstrap.cc:492] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.760725 23790 ts_tablet_manager.cc:1403] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.001s
I20251023 20:15:36.764335 23389 raft_consensus.cc:2468] T 473324b22da14618a84aa1136d081993 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:36.764698 25167 consensus_meta.cc:287] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.980s user 0.002s sys 0.001s
I20251023 20:15:36.764809 25167 raft_consensus.cc:515] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.765482 25167 leader_election.cc:290] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.769868 23470 leader_election.cc:304] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.770529 25282 ts_tablet_manager.cc:1929] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:36.770655 25282 log.cc:1199] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/9a4f012469fa4e8e827e485287ad76dd
I20251023 20:15:36.770823 25167 raft_consensus.cc:2804] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.770941 25167 raft_consensus.cc:697] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:36.771165 25167 consensus_queue.cc:237] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.782402 23786 tablet_bootstrap.cc:492] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.782644 23786 ts_tablet_manager.cc:1403] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.168s user 0.014s sys 0.020s
I20251023 20:15:36.783404 23786 raft_consensus.cc:359] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.783859 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 172) took 8004 ms. Trace:
I20251023 20:15:36.783967 24780 rpcz_store.cc:276] 1023 20:15:28.779315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.779361 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:36.521052 (+7741691us) ts_tablet_manager.cc:1151] Deleting tablet dafb9db710f94c328186e3179bf0b319
1023 20:15:36.521062 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.566899 (+ 45837us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.665740 (+ 98841us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.783830 (+118090us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19233,"tablet-delete.queue_time_us":7741529}
I20251023 20:15:36.784161 23214 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0" } new_table_name: "default.Table_2860d1804bda4f67b048ee52ba46f15e" modify_external_catalogs: true
I20251023 20:15:36.784319 24780 tablet_replica.cc:333] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:36.784509 24780 raft_consensus.cc:2243] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:36.784780 24780 raft_consensus.cc:2272] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:36.785367 24780 ts_tablet_manager.cc:1916] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:36.786284 23216 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_2bc741a605f74c0dbcc7aecc45137db4" } new_table_name: "default.Table_fb9b8f67f2a24061aea18ea29da5f332" modify_external_catalogs: true
I20251023 20:15:36.788300 23790 raft_consensus.cc:359] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.788604 23790 raft_consensus.cc:385] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.788698 23790 raft_consensus.cc:740] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.788968 23790 consensus_queue.cc:260] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.789530 23790 ts_tablet_manager.cc:1434] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20251023 20:15:36.789649 23786 raft_consensus.cc:385] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.789871 23790 tablet_bootstrap.cc:492] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.790083 23786 raft_consensus.cc:740] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.790562 23786 consensus_queue.cc:260] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.791916 23786 ts_tablet_manager.cc:1434] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251023 20:15:36.792457 23790 tablet_bootstrap.cc:654] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.793278 23786 tablet_bootstrap.cc:492] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.795713 23786 tablet_bootstrap.cc:654] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.797238 25235 raft_consensus.cc:493] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.797407 25235 raft_consensus.cc:515] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.798034 25235 leader_election.cc:290] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.799186 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19da816a483a4d5b9dc768ef697d54ec" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.800123 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea9be2a866cf46cfbec8db23a6d3f572" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:36.800259 23389 raft_consensus.cc:3060] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.802419 25167 raft_consensus.cc:493] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.802603 25167 raft_consensus.cc:515] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.803364 25167 leader_election.cc:290] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.804404 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 98a6a3425a49403e99c8ade114166e46 with delete_type TABLET_DATA_DELETED (replaced by tablet c6577d1bf92444f9a9ea1b5f5e7f005f) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.804876 25282 ts_tablet_manager.cc:1950] T 9a4f012469fa4e8e827e485287ad76dd P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:36.811458 23514 tablet_service.cc:1505] Processing CreateTablet for tablet 866f7e4cbed341b7b75182891c034c28 (DEFAULT_TABLE table=default.Table_6ea21ba3f1684634907e258298159baf [id=5962bfcedadc486d8f5e985d1c213fe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.812759 23514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866f7e4cbed341b7b75182891c034c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.814092 23786 tablet_bootstrap.cc:492] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.814224 23786 ts_tablet_manager.cc:1403] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20251023 20:15:36.815042 23786 raft_consensus.cc:359] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.815217 23786 raft_consensus.cc:385] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.815263 23786 raft_consensus.cc:740] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.815459 23786 consensus_queue.cc:260] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.815968 23786 ts_tablet_manager.cc:1434] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:36.816251 23786 tablet_bootstrap.cc:492] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.811458 23508 tablet_service.cc:1505] Processing CreateTablet for tablet c615066185c6472ba13bae91f72ee67c (DEFAULT_TABLE table=default.Table_39ef492038e649d1b5378fedd9cda522 [id=8ddeeaea0bf548eab7bef76cd980a311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.816527 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c615066185c6472ba13bae91f72ee67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.816488 23504 tablet_service.cc:1505] Processing CreateTablet for tablet cde03512dfe24dbdb2fb274affd558cf (DEFAULT_TABLE table=default.Table_a5e06ba2ba784868978a86ee8e607a2e [id=7718e9dc72d5495d8c9cb61bac6caf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.816767 23504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde03512dfe24dbdb2fb274affd558cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.818085 23786 tablet_bootstrap.cc:654] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.819725 23511 tablet_service.cc:1505] Processing CreateTablet for tablet ec4817cd994d41eaa6aee3dafab4d35e (DEFAULT_TABLE table=default.Table_68a2dbd9e4cb46cbab9b4b9e49e64d15 [id=5f0b47b3930349658b7c8dbc7d9978a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.822337 23511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4817cd994d41eaa6aee3dafab4d35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.816218 23497 tablet_service.cc:1505] Processing CreateTablet for tablet 6313bf6e463b42bd9990ec50904a3b60 (DEFAULT_TABLE table=default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0 [id=b9cb00bd8954475b909036fd9f123b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.825451 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dae906f257c346d7a1b28c9052d600b5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:36.825554 23333 leader_election.cc:343] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.825663 23497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6313bf6e463b42bd9990ec50904a3b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.825665 23399 raft_consensus.cc:3060] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.825668 23333 leader_election.cc:304] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.830708 25235 raft_consensus.cc:2749] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.831836 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet bc7793d1370d47ac8ee66d0246aba8fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.831918 25276 raft_consensus.cc:493] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.832059 25276 raft_consensus.cc:515] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.832451 23786 tablet_bootstrap.cc:492] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.832381 23787 tablet_bootstrap.cc:492] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.832576 23786 ts_tablet_manager.cc:1403] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251023 20:15:36.832615 23787 ts_tablet_manager.cc:1403] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.212s user 0.015s sys 0.031s
I20251023 20:15:36.833098 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dae906f257c346d7a1b28c9052d600b5" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:36.833250 23670 raft_consensus.cc:3060] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.833837 25276 leader_election.cc:290] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.833987 23786 raft_consensus.cc:359] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.834165 23786 raft_consensus.cc:385] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.834239 23786 raft_consensus.cc:740] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.834396 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b30e8ca2c96f4063a7f0e8f324c78b12" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.834455 23786 consensus_queue.cc:260] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.834908 23786 ts_tablet_manager.cc:1434] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.835203 23786 tablet_bootstrap.cc:492] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:36.835172 23604 leader_election.cc:343] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.835858 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b30e8ca2c96f4063a7f0e8f324c78b12" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.836386 23605 leader_election.cc:343] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.836479 23605 leader_election.cc:304] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.836838 23786 tablet_bootstrap.cc:654] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.836959 25657 raft_consensus.cc:2749] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.839669 25593 consensus_queue.cc:1048] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.842751 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.842947 23669 raft_consensus.cc:2468] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:36.843924 23470 leader_election.cc:304] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.844187 25167 raft_consensus.cc:2804] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.844277 25167 raft_consensus.cc:493] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.844401 25167 raft_consensus.cc:3060] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.844403 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3150a8478c144d285e999d5c9270fe0" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.844646 23667 raft_consensus.cc:2468] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:36.846310 23333 leader_election.cc:343] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.848659 25594 raft_consensus.cc:493] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.848794 25594 raft_consensus.cc:515] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.849331 25594 leader_election.cc:290] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.850807 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 92c50c4845524163834861438291a3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.851156 23639 tablet_service.cc:1552] Processing DeleteTablet for tablet 124703edbf08408a9ca69732674c0ad3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-23 20:15:36 UTC) from {username='slave'} at 127.0.0.1:60202
I20251023 20:15:36.851480 23644 tablet_service.cc:1505] Processing CreateTablet for tablet c615066185c6472ba13bae91f72ee67c (DEFAULT_TABLE table=default.Table_39ef492038e649d1b5378fedd9cda522 [id=8ddeeaea0bf548eab7bef76cd980a311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.852391 23787 raft_consensus.cc:359] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.852568 23787 raft_consensus.cc:385] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.852633 23787 raft_consensus.cc:740] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.852917 23787 consensus_queue.cc:260] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.853606 23787 ts_tablet_manager.cc:1434] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
W20251023 20:15:36.861212 23396 consensus_meta.cc:287] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.533s user 0.002s sys 0.000s
I20251023 20:15:36.861313 23396 raft_consensus.cc:2468] T 53f33020319d48c791e1156c67b55be6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:36.861636 25568 consensus_meta.cc:287] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.065s user 0.002s sys 0.001s
I20251023 20:15:36.861747 25568 raft_consensus.cc:515] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.862377 25568 leader_election.cc:290] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.868067 23644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c615066185c6472ba13bae91f72ee67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.871554 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4f012469fa4e8e827e485287ad76dd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
W20251023 20:15:36.872071 23470 leader_election.cc:343] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Illegal state: Tablet not RUNNING: SHUTDOWN
I20251023 20:15:36.874018 23648 tablet_service.cc:1505] Processing CreateTablet for tablet 866f7e4cbed341b7b75182891c034c28 (DEFAULT_TABLE table=default.Table_6ea21ba3f1684634907e258298159baf [id=5962bfcedadc486d8f5e985d1c213fe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.875692 23646 tablet_service.cc:1505] Processing CreateTablet for tablet cde03512dfe24dbdb2fb274affd558cf (DEFAULT_TABLE table=default.Table_a5e06ba2ba784868978a86ee8e607a2e [id=7718e9dc72d5495d8c9cb61bac6caf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.876070 23637 tablet_service.cc:1505] Processing CreateTablet for tablet ec4817cd994d41eaa6aee3dafab4d35e (DEFAULT_TABLE table=default.Table_68a2dbd9e4cb46cbab9b4b9e49e64d15 [id=5f0b47b3930349658b7c8dbc7d9978a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.878203 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:36.878791 23790 tablet_bootstrap.cc:492] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:36.879134 23639 tablet_service.cc:1505] Processing CreateTablet for tablet 6313bf6e463b42bd9990ec50904a3b60 (DEFAULT_TABLE table=default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0 [id=b9cb00bd8954475b909036fd9f123b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:36.880628 23786 tablet_bootstrap.cc:492] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.880772 23786 ts_tablet_manager.cc:1403] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20251023 20:15:36.880622 23790 ts_tablet_manager.cc:1403] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.000s
I20251023 20:15:36.881536 23786 raft_consensus.cc:359] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.881701 23786 raft_consensus.cc:385] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.881762 23786 raft_consensus.cc:740] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.881944 23786 consensus_queue.cc:260] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.882352 23786 ts_tablet_manager.cc:1434] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:36.882611 23786 tablet_bootstrap.cc:492] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.883976 23786 tablet_bootstrap.cc:654] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.887039 23787 tablet_bootstrap.cc:492] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.888134 23646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde03512dfe24dbdb2fb274affd558cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.887830 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 372) took 3360 ms. Trace:
I20251023 20:15:36.888332 25282 rpcz_store.cc:276] 1023 20:15:33.527554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.527633 (+ 79us) service_pool.cc:225] Handling call
1023 20:15:36.564801 (+3037168us) ts_tablet_manager.cc:1151] Deleting tablet 9a4f012469fa4e8e827e485287ad76dd
1023 20:15:36.564849 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.666684 (+101835us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.770383 (+103699us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.887811 (+117428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3033108}
I20251023 20:15:36.888681 25282 tablet_replica.cc:333] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:36.888864 25282 raft_consensus.cc:2243] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:36.888994 25282 raft_consensus.cc:2272] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:36.889060 23637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4817cd994d41eaa6aee3dafab4d35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.889266 23639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6313bf6e463b42bd9990ec50904a3b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.889533 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 9a4f012469fa4e8e827e485287ad76dd (table default.Table_e6e2be420a6342bc9d1311bd4fe6080c [id=8eeb69b1cebc42dcaae9f3306aa18013]) successfully deleted
I20251023 20:15:36.890880 23648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866f7e4cbed341b7b75182891c034c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:36.891423 23787 tablet_bootstrap.cc:654] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.891774 25282 ts_tablet_manager.cc:1916] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:36.892809 23786 tablet_bootstrap.cc:492] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.892930 23786 ts_tablet_manager.cc:1403] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:36.894125 23786 raft_consensus.cc:359] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.894320 23786 raft_consensus.cc:385] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.894397 23786 raft_consensus.cc:740] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:36.894676 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 48109.5us
I20251023 20:15:36.894642 23786 consensus_queue.cc:260] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.895179 23786 ts_tablet_manager.cc:1434] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:36.895467 23786 tablet_bootstrap.cc:492] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.897199 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet dafb9db710f94c328186e3179bf0b319 (table default.Table_82f6a78855d1488b9a8085c3bfbebab3 [id=f033b532f86d452c8f14e7f40ead8d62]) successfully deleted
I20251023 20:15:36.897379 23786 tablet_bootstrap.cc:654] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.902689 23215 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_519888eb86d3490baf10c0d615eea255" } modify_external_catalogs: true
I20251023 20:15:36.902875 23215 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55414:
table { table_name: "default.Table_519888eb86d3490baf10c0d615eea255" } modify_external_catalogs: true
I20251023 20:15:36.903422 25276 raft_consensus.cc:493] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.903548 25276 raft_consensus.cc:515] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.903621 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_f719c5f9fcda4036a855294384b2a635"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.904174 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f719c5f9fcda4036a855294384b2a635 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.907918 25276 leader_election.cc:290] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:36.908502 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:36.910995 23604 leader_election.cc:343] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.913257 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:36.913807 23605 leader_election.cc:343] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.913918 23605 leader_election.cc:304] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.914536 25276 raft_consensus.cc:2749] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.914736 23786 tablet_bootstrap.cc:492] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.914896 23786 ts_tablet_manager.cc:1403] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.007s
I20251023 20:15:36.915624 23786 raft_consensus.cc:359] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.915817 23786 raft_consensus.cc:385] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.915899 23786 raft_consensus.cc:740] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.916090 23786 consensus_queue.cc:260] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.916742 23786 ts_tablet_manager.cc:1434] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:36.917146 23786 tablet_bootstrap.cc:492] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.919039 23786 tablet_bootstrap.cc:654] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.924024 23787 tablet_bootstrap.cc:492] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.924149 23787 ts_tablet_manager.cc:1403] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20251023 20:15:36.924844 23787 raft_consensus.cc:359] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.924993 23787 raft_consensus.cc:385] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.925036 23787 raft_consensus.cc:740] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.925258 23787 consensus_queue.cc:260] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.925343 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.925746 23787 ts_tablet_manager.cc:1434] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:36.926023 23787 tablet_bootstrap.cc:492] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.926141 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.926299 23665 raft_consensus.cc:2468] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:36.927618 23787 tablet_bootstrap.cc:654] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.928655 25454 consensus_queue.cc:1048] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.933502 25657 raft_consensus.cc:493] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.933663 25657 raft_consensus.cc:515] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.934273 25657 leader_election.cc:290] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:36.934379 23471 leader_election.cc:343] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.934851 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cdde6b251b14df4bf44142ce263b84a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.935024 23534 raft_consensus.cc:2468] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 0.
I20251023 20:15:36.935091 25454 consensus_queue.cc:1048] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:36.935303 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cdde6b251b14df4bf44142ce263b84a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.935379 23604 leader_election.cc:304] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.935635 25657 raft_consensus.cc:2804] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.935714 25657 raft_consensus.cc:493] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.935825 25657 raft_consensus.cc:3060] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:36.936254 23605 leader_election.cc:343] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.939926 25235 raft_consensus.cc:493] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.940083 25235 raft_consensus.cc:515] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.940735 25235 leader_election.cc:290] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.944921 23526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:36.908455 23790 raft_consensus.cc:359] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.945889 23790 raft_consensus.cc:385] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.946074 23790 raft_consensus.cc:740] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.946512 25454 raft_consensus.cc:493] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.946667 25454 raft_consensus.cc:515] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.947237 25454 leader_election.cc:290] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.947744 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.948006 23665 raft_consensus.cc:2468] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:36.955713 25276 raft_consensus.cc:493] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:36.956063 25276 raft_consensus.cc:515] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.956882 25276 leader_election.cc:290] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:36.957031 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa0a46ffa626447eab4cdb956e798005" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:36.957222 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e5720c96730425dafb550094cf936cd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:36.957389 23786 tablet_bootstrap.cc:492] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.957513 23786 ts_tablet_manager.cc:1403] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.002s
I20251023 20:15:36.957798 23666 raft_consensus.cc:2468] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:36.958115 23526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e5720c96730425dafb550094cf936cd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:36.958204 23605 leader_election.cc:343] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.958266 23786 raft_consensus.cc:359] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.958449 23786 raft_consensus.cc:385] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.958526 23786 raft_consensus.cc:740] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:36.958562 23604 leader_election.cc:343] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:36.958683 23604 leader_election.cc:304] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:36.958784 23786 consensus_queue.cc:260] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.959072 25276 raft_consensus.cc:2749] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:36.959414 23786 ts_tablet_manager.cc:1434] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:36.959493 23470 leader_election.cc:304] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:36.959728 25470 raft_consensus.cc:2804] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:36.959794 25470 raft_consensus.cc:493] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:36.959849 25470 raft_consensus.cc:3060] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:36.960145 23786 tablet_bootstrap.cc:492] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.961710 23786 tablet_bootstrap.cc:654] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.963476 24780 ts_tablet_manager.cc:1929] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:36.963593 24780 log.cc:1199] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d62dbf044acd4459b26d2e7e50f6206e
I20251023 20:15:36.964054 24780 ts_tablet_manager.cc:1950] T d62dbf044acd4459b26d2e7e50f6206e P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:36.964195 25172 raft_consensus.cc:515] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.964829 25172 leader_election.cc:290] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:36.965165 25462 consensus_meta.cc:287] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.116s user 0.002s sys 0.000s
I20251023 20:15:36.947434 23790 consensus_queue.cc:260] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.965292 25462 raft_consensus.cc:515] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.965865 25462 leader_election.cc:290] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:36.966035 23787 tablet_bootstrap.cc:492] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.966200 23670 raft_consensus.cc:2468] T dae906f257c346d7a1b28c9052d600b5 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:36.966466 23790 ts_tablet_manager.cc:1434] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.085s user 0.001s sys 0.001s
I20251023 20:15:36.966746 23790 tablet_bootstrap.cc:492] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:36.966200 23787 ts_tablet_manager.cc:1403] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20251023 20:15:36.967944 23787 raft_consensus.cc:359] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.968116 23787 raft_consensus.cc:385] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.968190 23787 raft_consensus.cc:740] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.968389 23787 consensus_queue.cc:260] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.968967 23787 ts_tablet_manager.cc:1434] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:36.969238 23787 tablet_bootstrap.cc:492] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:36.969923 23786 tablet_bootstrap.cc:492] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.970080 23786 ts_tablet_manager.cc:1403] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20251023 20:15:36.970855 23786 raft_consensus.cc:359] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.971043 23786 raft_consensus.cc:385] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.971112 23786 raft_consensus.cc:740] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.971330 23786 consensus_queue.cc:260] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.971812 23786 ts_tablet_manager.cc:1434] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:36.971978 23533 raft_consensus.cc:2393] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:36.972187 23786 tablet_bootstrap.cc:492] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.974634 23786 tablet_bootstrap.cc:654] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.976363 23787 tablet_bootstrap.cc:654] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.982756 23786 tablet_bootstrap.cc:492] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:36.982957 23786 ts_tablet_manager.cc:1403] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20251023 20:15:36.986359 23470 leader_election.cc:304] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:36.987474 23470 leader_election.cc:343] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 39a62c97e0a3459e91c24c1c2f92389a
I20251023 20:15:36.988046 25462 raft_consensus.cc:2804] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:36.988209 25462 raft_consensus.cc:697] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:36.988449 25462 consensus_queue.cc:237] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.988525 23786 raft_consensus.cc:359] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.988682 23786 raft_consensus.cc:385] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:36.988763 23786 raft_consensus.cc:740] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:36.988935 23786 consensus_queue.cc:260] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:36.989492 23786 ts_tablet_manager.cc:1434] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251023 20:15:36.989758 23786 tablet_bootstrap.cc:492] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:36.990414 23210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55414:
name: "default.Table_0984ad1922884cb19d48dc21a903dfe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251023 20:15:36.991008 23210 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0984ad1922884cb19d48dc21a903dfe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251023 20:15:36.992213 23786 tablet_bootstrap.cc:654] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:36.993731 23787 tablet_bootstrap.cc:492] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:36.993906 23787 ts_tablet_manager.cc:1403] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251023 20:15:37.006057 23333 leader_election.cc:304] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.007489 25489 raft_consensus.cc:2804] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.007582 25489 raft_consensus.cc:493] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.007644 25489 raft_consensus.cc:3060] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.009431 23790 tablet_bootstrap.cc:654] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.011023 23333 leader_election.cc:304] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.011260 25595 raft_consensus.cc:2804] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.011371 25595 raft_consensus.cc:493] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.011471 25595 raft_consensus.cc:3060] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.012404 23333 leader_election.cc:304] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.016368 23787 raft_consensus.cc:359] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.016564 23787 raft_consensus.cc:385] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.016621 23787 raft_consensus.cc:740] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.016861 25235 raft_consensus.cc:2804] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.016819 23787 consensus_queue.cc:260] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.016961 25235 raft_consensus.cc:493] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.017026 25235 raft_consensus.cc:3060] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:37.017360 23333 consensus_peers.cc:597] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72408899f710470baaf6fdb268cb04c8. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:37.017387 23787 ts_tablet_manager.cc:1434] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.023s user 0.016s sys 0.000s
I20251023 20:15:37.017688 23787 tablet_bootstrap.cc:492] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.018376 23371 tablet_service.cc:1505] Processing CreateTablet for tablet c615066185c6472ba13bae91f72ee67c (DEFAULT_TABLE table=default.Table_39ef492038e649d1b5378fedd9cda522 [id=8ddeeaea0bf548eab7bef76cd980a311]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:37.018774 23371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c615066185c6472ba13bae91f72ee67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:37.018977 23787 tablet_bootstrap.cc:654] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.020812 23366 tablet_service.cc:1505] Processing CreateTablet for tablet 866f7e4cbed341b7b75182891c034c28 (DEFAULT_TABLE table=default.Table_6ea21ba3f1684634907e258298159baf [id=5962bfcedadc486d8f5e985d1c213fe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:37.021853 23375 tablet_service.cc:1505] Processing CreateTablet for tablet 6313bf6e463b42bd9990ec50904a3b60 (DEFAULT_TABLE table=default.Table_031cfa89b96e4bd8b0b74307d2fb1bd0 [id=b9cb00bd8954475b909036fd9f123b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:37.022225 23375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6313bf6e463b42bd9990ec50904a3b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:37.025166 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a4f012469fa4e8e827e485287ad76dd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.025605 23362 tablet_service.cc:1505] Processing CreateTablet for tablet cde03512dfe24dbdb2fb274affd558cf (DEFAULT_TABLE table=default.Table_a5e06ba2ba784868978a86ee8e607a2e [id=7718e9dc72d5495d8c9cb61bac6caf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:37.025941 23362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde03512dfe24dbdb2fb274affd558cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:37.027217 23787 tablet_bootstrap.cc:492] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.027328 23787 ts_tablet_manager.cc:1403] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20251023 20:15:37.027921 23787 raft_consensus.cc:359] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.028064 23787 raft_consensus.cc:385] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.028102 23787 raft_consensus.cc:740] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.028272 23787 consensus_queue.cc:260] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.028716 23787 ts_tablet_manager.cc:1434] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:37.028848 23372 tablet_service.cc:1505] Processing CreateTablet for tablet ec4817cd994d41eaa6aee3dafab4d35e (DEFAULT_TABLE table=default.Table_68a2dbd9e4cb46cbab9b4b9e49e64d15 [id=5f0b47b3930349658b7c8dbc7d9978a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251023 20:15:37.028971 23787 tablet_bootstrap.cc:492] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.029156 23372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4817cd994d41eaa6aee3dafab4d35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:37.030251 23787 tablet_bootstrap.cc:654] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.030778 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39a62c97e0a3459e91c24c1c2f92389a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.030998 23393 raft_consensus.cc:2393] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.033536 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.033847 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa0a46ffa626447eab4cdb956e798005" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.033994 23787 tablet_bootstrap.cc:492] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.033988 23392 raft_consensus.cc:2468] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.034101 23787 ts_tablet_manager.cc:1403] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20251023 20:15:37.034343 23366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866f7e4cbed341b7b75182891c034c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251023 20:15:37.034772 23787 raft_consensus.cc:359] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.034910 23787 raft_consensus.cc:385] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.034950 23787 raft_consensus.cc:740] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.035120 23787 consensus_queue.cc:260] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.035523 23787 ts_tablet_manager.cc:1434] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:37.035773 23787 tablet_bootstrap.cc:492] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.037002 23787 tablet_bootstrap.cc:654] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.039366 23526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.040050 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.040215 23665 raft_consensus.cc:2468] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.043059 23786 tablet_bootstrap.cc:492] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.043231 23786 ts_tablet_manager.cc:1403] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.004s
I20251023 20:15:37.043859 23533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d22aa40e5844d78a8679a3ad1b08da9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.044090 23533 raft_consensus.cc:3060] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.044060 23786 raft_consensus.cc:359] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.044255 23786 raft_consensus.cc:385] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.044324 23786 raft_consensus.cc:740] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.044531 23786 consensus_queue.cc:260] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.044967 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d22aa40e5844d78a8679a3ad1b08da9" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.045056 23786 ts_tablet_manager.cc:1434] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:37.045130 23665 raft_consensus.cc:3060] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.045368 23786 tablet_bootstrap.cc:492] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.046964 23786 tablet_bootstrap.cc:654] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.052796 23790 tablet_bootstrap.cc:492] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.052971 23790 ts_tablet_manager.cc:1403] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20251023 20:15:37.053730 23790 raft_consensus.cc:359] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.053946 23790 raft_consensus.cc:385] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.054021 23790 raft_consensus.cc:740] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.054227 23790 consensus_queue.cc:260] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.054690 23790 ts_tablet_manager.cc:1434] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:37.054977 23790 tablet_bootstrap.cc:492] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.056732 23790 tablet_bootstrap.cc:654] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.057420 25462 consensus_queue.cc:1048] T 29c2c976d6a3440394f478a31abc7134 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:37.060395 23471 leader_election.cc:343] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.066854 23387 raft_consensus.cc:2468] T e658aa360e524a9f9bdb35c1f68ae426 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:37.067293 25554 consensus_meta.cc:287] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.127s user 0.003s sys 0.000s
I20251023 20:15:37.067414 25554 raft_consensus.cc:515] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.068033 25554 leader_election.cc:290] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.068351 25657 raft_consensus.cc:515] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.069001 25657 leader_election.cc:290] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.069373 25282 ts_tablet_manager.cc:1929] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:37.069468 25282 log.cc:1199] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/453eea5cdd2e47f29577c61b40ee59c0
I20251023 20:15:37.069741 25282 ts_tablet_manager.cc:1950] T 453eea5cdd2e47f29577c61b40ee59c0 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:37.073964 23787 tablet_bootstrap.cc:492] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.074086 23787 ts_tablet_manager.cc:1403] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.014s
I20251023 20:15:37.074714 23787 raft_consensus.cc:359] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.074877 23787 raft_consensus.cc:385] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.074925 23787 raft_consensus.cc:740] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.075102 23787 consensus_queue.cc:260] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.075549 23787 ts_tablet_manager.cc:1434] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:37.075812 23787 tablet_bootstrap.cc:492] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.077189 23787 tablet_bootstrap.cc:654] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.081809 23786 tablet_bootstrap.cc:492] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.081964 23786 ts_tablet_manager.cc:1403] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.013s
I20251023 20:15:37.082983 23786 raft_consensus.cc:359] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.083151 23786 raft_consensus.cc:385] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.083231 23786 raft_consensus.cc:740] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.083413 23786 consensus_queue.cc:260] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.083669 25454 raft_consensus.cc:493] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.083788 25454 raft_consensus.cc:515] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.084347 25454 leader_election.cc:290] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.085135 23786 ts_tablet_manager.cc:1434] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:37.085407 23786 tablet_bootstrap.cc:492] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.087575 23786 tablet_bootstrap.cc:654] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.088853 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1efa4fcaeb049b484ca59b12025ce9a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.089041 23666 raft_consensus.cc:2468] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.089582 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cdde6b251b14df4bf44142ce263b84a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.088830 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e343a7d39cf4329b99c9f9ffd8cefb2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.090059 23670 raft_consensus.cc:3060] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:37.090116 23605 leader_election.cc:343] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.092540 23470 leader_election.cc:304] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.092692 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cdde6b251b14df4bf44142ce263b84a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.060582 23471 leader_election.cc:304] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:37.092883 25462 raft_consensus.cc:2804] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.093055 25462 raft_consensus.cc:493] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.093127 25554 raft_consensus.cc:2749] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.093366 25462 raft_consensus.cc:3060] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.093429 23787 tablet_bootstrap.cc:492] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.093629 23787 ts_tablet_manager.cc:1403] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.008s
I20251023 20:15:37.092881 23534 raft_consensus.cc:3060] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.098439 23787 raft_consensus.cc:359] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.098592 23787 raft_consensus.cc:385] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.098641 23787 raft_consensus.cc:740] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.098822 23787 consensus_queue.cc:260] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.099287 23787 ts_tablet_manager.cc:1434] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20251023 20:15:37.099884 23787 tablet_bootstrap.cc:492] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.101202 25554 raft_consensus.cc:493] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.101390 23787 tablet_bootstrap.cc:654] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.101325 25554 raft_consensus.cc:515] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.101883 25554 leader_election.cc:290] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.102823 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.103009 23666 raft_consensus.cc:2468] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.103475 23470 leader_election.cc:304] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.103754 25554 raft_consensus.cc:2804] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.103922 25554 raft_consensus.cc:493] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.104008 25554 raft_consensus.cc:3060] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.105588 25454 raft_consensus.cc:493] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.105731 25454 raft_consensus.cc:515] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.106238 25454 leader_election.cc:290] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.107323 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb7935c4cf64e2f802ca12c1f63317b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
W20251023 20:15:37.107415 23333 leader_election.cc:343] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.107492 23666 raft_consensus.cc:2468] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.107805 23333 leader_election.cc:304] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.108058 25594 raft_consensus.cc:2749] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.109567 23470 leader_election.cc:304] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.111950 25454 raft_consensus.cc:2804] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.112038 25454 raft_consensus.cc:493] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.112923 25454 raft_consensus.cc:3060] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.121591 23471 leader_election.cc:304] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:37.122224 25568 raft_consensus.cc:2749] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.123478 25568 consensus_queue.cc:1048] T 1b220ceff0924b26ae8a54dfb8705456 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.128036 23212 catalog_manager.cc:5649] T c6f3abcaa81c4646bc1432bcbd1d4fcf P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: HEALTHY } } }
I20251023 20:15:37.128827 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e343a7d39cf4329b99c9f9ffd8cefb2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.128998 23398 raft_consensus.cc:3060] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.129261 23388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb7935c4cf64e2f802ca12c1f63317b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.129335 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1efa4fcaeb049b484ca59b12025ce9a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.129472 23392 raft_consensus.cc:2468] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.129496 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.130167 23333 leader_election.cc:304] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.130506 25172 raft_consensus.cc:2804] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.130616 25172 raft_consensus.cc:493] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.130688 25172 raft_consensus.cc:3060] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.137751 23787 tablet_bootstrap.cc:492] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.138533 23787 ts_tablet_manager.cc:1403] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.009s
I20251023 20:15:37.139429 23787 raft_consensus.cc:359] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.140448 23787 raft_consensus.cc:385] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.140655 23787 raft_consensus.cc:740] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.140820 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c6f3abcaa81c4646bc1432bcbd1d4fcf (table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]). Expected version 2 got 0
I20251023 20:15:37.141163 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]). Expected version 1 got 0
I20251023 20:15:37.141172 23787 consensus_queue.cc:260] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.142163 23787 ts_tablet_manager.cc:1434] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:37.142895 23787 tablet_bootstrap.cc:492] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.143573 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a744214a24cc4bf28654ffa417ce0d3d (table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]). Expected version 1 got 0
I20251023 20:15:37.143925 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet e5af107f9a144edb9080b7135ab8aeb9 (table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]). Expected version 1 got 0
I20251023 20:15:37.144201 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]). Expected version 1 got 0
W20251023 20:15:37.144200 23471 leader_election.cc:343] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:37.144694 23471 leader_election.cc:343] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.144817 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet eb5a6d9a4509431f97b69a3b138b5829 (table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]). Expected version 2 got 0
I20251023 20:15:37.145205 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 766882a030b446eb9406b3e32d903ee5 (table default.Table_511899f307ee49fb8612942d4b966ee2 [id=12648eb0bb8b4909814e3f2cb9cec1ae]). Expected version 1 got 0
W20251023 20:15:37.145352 23333 consensus_peers.cc:597] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:37.145975 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]). Expected version 2 got 0
I20251023 20:15:37.149693 23790 tablet_bootstrap.cc:492] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.149869 23790 ts_tablet_manager.cc:1403] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.000s
I20251023 20:15:37.150638 23790 raft_consensus.cc:359] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.150843 23790 raft_consensus.cc:385] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.150925 23790 raft_consensus.cc:740] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.151165 23790 consensus_queue.cc:260] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.151645 23790 ts_tablet_manager.cc:1434] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:37.151923 23790 tablet_bootstrap.cc:492] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.153530 23790 tablet_bootstrap.cc:654] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.156898 23787 tablet_bootstrap.cc:654] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.158944 23786 tablet_bootstrap.cc:492] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.159124 23786 ts_tablet_manager.cc:1403] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.074s user 0.000s sys 0.005s
I20251023 20:15:37.159869 23786 raft_consensus.cc:359] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.160081 23786 raft_consensus.cc:385] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.160162 23786 raft_consensus.cc:740] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.160936 23786 consensus_queue.cc:260] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.162031 23786 ts_tablet_manager.cc:1434] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251023 20:15:37.164139 25657 raft_consensus.cc:493] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.164268 25657 raft_consensus.cc:515] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.164796 25657 leader_election.cc:290] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.165411 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.165887 23605 leader_election.cc:343] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.169970 25466 raft_consensus.cc:493] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.174356 25594 raft_consensus.cc:493] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.174485 25594 raft_consensus.cc:515] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.175055 25594 leader_election.cc:290] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.175505 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "225ce6aa52c54f76bd56247f01219dbc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.175694 23666 raft_consensus.cc:2468] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.176553 23333 leader_election.cc:304] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.176795 24821 raft_consensus.cc:2804] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.176887 24821 raft_consensus.cc:493] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.176947 24821 raft_consensus.cc:3060] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.178391 25657 raft_consensus.cc:493] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.178526 25657 raft_consensus.cc:515] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.179045 25657 leader_election.cc:290] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.179688 23535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4798f6eea5745ba8a0d4b8cb173cc5c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.180331 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4798f6eea5745ba8a0d4b8cb173cc5c" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.180855 23604 leader_election.cc:343] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:37.181247 23605 leader_election.cc:343] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.181339 23605 leader_election.cc:304] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.181756 25657 raft_consensus.cc:2749] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.187407 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet efad9b4a8e2e446f917df7ca26fbf789 (table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]). Expected version 1 got 0
I20251023 20:15:37.187770 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 559a70cf17ad4145acdb3033e01ad217 (table default.Table_12e6cb242b54467f9041afc60fa47eb3 [id=91a585d91b924838996b49fe349985f0]). Expected version 1 got 0
I20251023 20:15:37.193473 23212 catalog_manager.cc:5649] T 92576756ec574ceaa4c81f435cb72301 P d220aba7c9c0400398ffde917376ee48 reported cstate change: term changed from 0 to 1, leader changed from <none> to d220aba7c9c0400398ffde917376ee48 (127.22.127.129). New cstate: current_term: 1 leader_uuid: "d220aba7c9c0400398ffde917376ee48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } health_report { overall_health: UNKNOWN } } }
I20251023 20:15:37.194092 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]). Expected version 1 got 0
I20251023 20:15:37.194294 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3a216aeebe649f79b6d873e1c59df37 (table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]). Expected version 1 got 0
I20251023 20:15:37.194494 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a4d71f0cfdb6449fbd74b854c15326d6 (table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]). Expected version 2 got 0
I20251023 20:15:37.194980 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet ea9be2a866cf46cfbec8db23a6d3f572 (table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]). Expected version 1 got 0
I20251023 20:15:37.195924 23790 tablet_bootstrap.cc:492] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.196099 23790 ts_tablet_manager.cc:1403] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20251023 20:15:37.196997 23790 raft_consensus.cc:359] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.197227 23790 raft_consensus.cc:385] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.197301 23790 raft_consensus.cc:740] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.198474 23790 consensus_queue.cc:260] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.198729 23787 tablet_bootstrap.cc:492] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.198907 23787 ts_tablet_manager.cc:1403] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.003s
I20251023 20:15:37.199030 23790 ts_tablet_manager.cc:1434] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:37.199309 23790 tablet_bootstrap.cc:492] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.201249 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet fa0a46ffa626447eab4cdb956e798005 (table default.Table_3c3fd30fc677490da11231e62dd177aa [id=09f8335a5b03409c90559c8313f5d135]). Expected version 1 got 0
I20251023 20:15:37.201659 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet abdad7d76eb8441496d38642382f74bb (table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]). Expected version 4 got 0
I20251023 20:15:37.202052 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]). Expected version 1 got 0
I20251023 20:15:37.202236 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]). Expected version 2 got 0
I20251023 20:15:37.202631 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cd548fd9de284377ab557bd98f621068 (table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]). Expected version 1 got 0
I20251023 20:15:37.203429 23790 tablet_bootstrap.cc:654] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.204164 23787 raft_consensus.cc:359] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.204222 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]). Expected version 2 got 0
I20251023 20:15:37.204365 23787 raft_consensus.cc:385] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.204447 23787 raft_consensus.cc:740] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.204689 23787 consensus_queue.cc:260] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.205027 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]). Expected version 2 got 0
I20251023 20:15:37.207018 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:37.207865 23604 leader_election.cc:343] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.207975 23604 leader_election.cc:304] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.208324 25657 raft_consensus.cc:2749] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.209357 25465 consensus_queue.cc:1048] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20251023 20:15:37.209565 23787 ts_tablet_manager.cc:1434] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251023 20:15:37.210165 23787 tablet_bootstrap.cc:492] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.210088 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "225ce6aa52c54f76bd56247f01219dbc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.210363 23527 raft_consensus.cc:2468] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.210541 25466 raft_consensus.cc:515] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.211259 25466 leader_election.cc:290] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.212131 23787 tablet_bootstrap.cc:654] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.212409 25657 raft_consensus.cc:493] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.212574 25657 raft_consensus.cc:515] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.212653 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa59141039de402e9ccce427604ecb31" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.212834 23666 raft_consensus.cc:2468] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.212999 25465 consensus_queue.cc:1048] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251023 20:15:37.213120 25657 leader_election.cc:290] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.213317 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa59141039de402e9ccce427604ecb31" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.213922 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb2bf696d2f4b639dfb0eff2a65e6f8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.214601 23605 leader_election.cc:343] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.215104 23470 leader_election.cc:304] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:37.215323 23471 leader_election.cc:343] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.216862 23786 tablet_bootstrap.cc:492] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.217849 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb2bf696d2f4b639dfb0eff2a65e6f8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.220557 23786 tablet_bootstrap.cc:654] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:37.221347 23604 leader_election.cc:343] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.221458 23604 leader_election.cc:304] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.221791 25657 raft_consensus.cc:2749] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.222402 25465 raft_consensus.cc:2804] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.222506 25465 raft_consensus.cc:493] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.222591 25465 raft_consensus.cc:3060] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.228044 23786 tablet_bootstrap.cc:492] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.228200 23786 ts_tablet_manager.cc:1403] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251023 20:15:37.228955 23786 raft_consensus.cc:359] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.229151 23786 raft_consensus.cc:385] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.229274 23786 raft_consensus.cc:740] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.229506 23786 consensus_queue.cc:260] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.230079 23786 ts_tablet_manager.cc:1434] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:37.230480 23786 tablet_bootstrap.cc:492] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.231704 25715 consensus_queue.cc:1048] T 27b593b020b34a969f77d2f68178e89d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.232278 23786 tablet_bootstrap.cc:654] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.235361 23787 tablet_bootstrap.cc:492] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.235532 23787 ts_tablet_manager.cc:1403] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20251023 20:15:37.236238 23787 raft_consensus.cc:359] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.236428 23787 raft_consensus.cc:385] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.236508 23787 raft_consensus.cc:740] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.236742 23787 consensus_queue.cc:260] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.237363 23787 ts_tablet_manager.cc:1434] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:37.237643 23787 tablet_bootstrap.cc:492] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.238389 25619 raft_consensus.cc:493] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.238540 25619 raft_consensus.cc:515] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.239190 25619 leader_election.cc:290] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.241796 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b446550686024c3ca80f106112a2110f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.241986 23391 raft_consensus.cc:2468] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.242416 23471 leader_election.cc:304] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.242653 25619 raft_consensus.cc:2804] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.242769 25619 raft_consensus.cc:493] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.242733 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b446550686024c3ca80f106112a2110f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.242842 25619 raft_consensus.cc:3060] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.242894 23666 raft_consensus.cc:2468] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.249439 25657 raft_consensus.cc:493] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.249565 25657 raft_consensus.cc:515] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.250100 25657 leader_election.cc:290] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.251184 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88f1df552f93427e91ca19c8170d69bd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.251686 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88f1df552f93427e91ca19c8170d69bd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.252389 25592 raft_consensus.cc:493] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.252509 25592 raft_consensus.cc:515] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.253140 25592 leader_election.cc:290] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.253944 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db758a7a7a89410f8ac1da72048fbc56" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.254118 23527 raft_consensus.cc:2468] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.254455 23333 leader_election.cc:304] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.254702 25592 raft_consensus.cc:2804] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.254781 25592 raft_consensus.cc:493] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.254863 25592 raft_consensus.cc:3060] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.255154 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db758a7a7a89410f8ac1da72048fbc56" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.255302 23666 raft_consensus.cc:2468] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:37.255650 23604 leader_election.cc:343] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:37.255932 23605 leader_election.cc:343] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.256027 23605 leader_election.cc:304] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.256345 25657 raft_consensus.cc:2749] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.269032 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 375) took 3740 ms. Trace:
I20251023 20:15:37.269187 25282 rpcz_store.cc:276] 1023 20:15:33.528839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.528899 (+ 60us) service_pool.cc:225] Handling call
1023 20:15:36.888643 (+3359744us) ts_tablet_manager.cc:1151] Deleting tablet 453eea5cdd2e47f29577c61b40ee59c0
1023 20:15:36.888652 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.978149 (+ 89497us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.069257 (+ 91108us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.269004 (+199747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3356924}
I20251023 20:15:37.269626 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 376) took 3740 ms. Trace:
I20251023 20:15:37.269697 25282 rpcz_store.cc:276] 1023 20:15:33.529335 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.529383 (+ 48us) service_pool.cc:225] Handling call
1023 20:15:37.269518 (+3740135us) ts_tablet_manager.cc:1151] Deleting tablet d9f73477a06542c0b215ae86716cdc4e
1023 20:15:37.269530 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.269612 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3737759}
I20251023 20:15:37.269975 25282 tablet_replica.cc:333] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:37.270166 25282 raft_consensus.cc:2243] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:37.270308 25282 raft_consensus.cc:2272] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:37.269196 25657 raft_consensus.cc:493] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.270951 25657 raft_consensus.cc:515] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.271602 25657 leader_election.cc:290] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.272313 25282 ts_tablet_manager.cc:1916] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:37.272532 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8317a54a5f7b4afe8da9d336268efe43" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.273319 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8317a54a5f7b4afe8da9d336268efe43" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:37.273380 23605 leader_election.cc:343] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:37.273850 23604 leader_election.cc:343] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.273964 23604 leader_election.cc:304] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.274170 25657 raft_consensus.cc:2749] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.257901 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]). Expected version 1 got 0
I20251023 20:15:37.270802 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 453eea5cdd2e47f29577c61b40ee59c0 (table default.Table_8c22652456c84553954550e1018e83e5 [id=ce7dc9691f2a4eeca898b4bf501fae49]) successfully deleted
W20251023 20:15:37.277433 23204 catalog_manager.cc:4952] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): delete failed for tablet d9f73477a06542c0b215ae86716cdc4e because the tablet was not found. No further retry: Not found: Tablet not found: d9f73477a06542c0b215ae86716cdc4e
I20251023 20:15:37.277967 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]). Expected version 1 got 0
I20251023 20:15:37.279441 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]). Expected version 2 got 0
I20251023 20:15:37.281883 23790 tablet_bootstrap.cc:492] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.282096 23790 ts_tablet_manager.cc:1403] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20251023 20:15:37.282990 23790 raft_consensus.cc:359] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.287381 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]). Expected version 1 got 0
I20251023 20:15:37.287758 23790 raft_consensus.cc:385] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.288224 23790 raft_consensus.cc:740] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.288683 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]). Expected version 1 got 0
I20251023 20:15:37.291990 23790 consensus_queue.cc:260] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.292716 23790 ts_tablet_manager.cc:1434] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:37.293109 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]). Expected version 1 got 0
I20251023 20:15:37.293740 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]). Expected version 1 got 0
I20251023 20:15:37.294337 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet a570f339890946b7ab20c15de68462d5 (table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]). Expected version 1 got 0
I20251023 20:15:37.294526 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 8e718cb7205643fd8c3045c3fab30d3e (table default.Table_3eeb50ae897f4006ad63033f2ad0324a [id=8477c10fb2f4422996d4eab8717d7fe8]). Expected version 1 got 0
I20251023 20:15:37.295708 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]). Expected version 1 got 0
I20251023 20:15:37.296159 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_7366a659fef84d17925d61af9a4d456c [id=910ad9776933488eb34806b4aaff92d8]). Expected version 2 got 1
I20251023 20:15:37.297305 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet cf38936bb12846bd81a2c79ffbd18ccb (table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]). Expected version 1 got 0
I20251023 20:15:37.299080 25657 raft_consensus.cc:493] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.299230 25657 raft_consensus.cc:515] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.299852 25657 leader_election.cc:290] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.300537 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "255521d943d34713aaf773b15f9168c6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:37.301131 23604 leader_election.cc:343] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.301398 25715 raft_consensus.cc:493] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.301630 25715 raft_consensus.cc:515] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.302377 25715 leader_election.cc:290] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.303157 23470 leader_election.cc:343] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: b90810781f724c78935a33d868ffa27d
I20251023 20:15:37.310742 25568 consensus_queue.cc:1048] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.311841 23038 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu pid 23185
I20251023 20:15:37.319501 23790 tablet_bootstrap.cc:492] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.321422 23790 tablet_bootstrap.cc:654] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.322476 23786 tablet_bootstrap.cc:492] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.322687 23786 ts_tablet_manager.cc:1403] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.092s user 0.017s sys 0.033s
I20251023 20:15:37.323400 23786 raft_consensus.cc:359] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.323637 23786 raft_consensus.cc:385] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.323732 23786 raft_consensus.cc:740] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.323987 23786 consensus_queue.cc:260] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.324522 23786 ts_tablet_manager.cc:1434] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:37.324846 23786 tablet_bootstrap.cc:492] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.326644 23786 tablet_bootstrap.cc:654] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.313122 23249 generic_service.cc:196] Checking for leaks (request via RPC)
I20251023 20:15:37.333145 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]). Expected version 1 got 0
W20251023 20:15:37.345922 23471 leader_election.cc:343] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: b90810781f724c78935a33d868ffa27d
I20251023 20:15:37.346113 23471 leader_election.cc:304] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:37.346385 25715 raft_consensus.cc:2749] T b90810781f724c78935a33d868ffa27d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251023 20:15:37.346853 25599 consensus_meta.cc:287] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.154s user 0.002s sys 0.000s
I20251023 20:15:37.346967 25599 raft_consensus.cc:515] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.347642 25599 leader_election.cc:290] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.347923 23670 raft_consensus.cc:2468] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.348426 23470 leader_election.cc:304] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.348956 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3befdfc85bca42bb816e11d1dd832c8c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.349125 23670 raft_consensus.cc:3060] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.349679 23787 tablet_bootstrap.cc:654] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:37.351729 25717 log.cc:927] Time spent T 27b593b020b34a969f77d2f68178e89d P d220aba7c9c0400398ffde917376ee48: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
W20251023 20:15:37.352152 24297 consensus_meta.cc:287] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.738s user 0.000s sys 0.003s
I20251023 20:15:37.352267 24297 raft_consensus.cc:515] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.354015 25568 raft_consensus.cc:2804] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.354180 25568 raft_consensus.cc:697] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.354446 25568 consensus_queue.cc:237] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.355675 24297 leader_election.cc:290] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.361482 23786 tablet_bootstrap.cc:492] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.361634 23786 ts_tablet_manager.cc:1403] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.000s
I20251023 20:15:37.362462 23786 raft_consensus.cc:359] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.362633 23786 raft_consensus.cc:385] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.362710 23786 raft_consensus.cc:740] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.362919 23786 consensus_queue.cc:260] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.363406 23786 ts_tablet_manager.cc:1434] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:37.363668 23786 tablet_bootstrap.cc:492] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:37.366030 23389 consensus_meta.cc:287] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.566s user 0.003s sys 0.000s
W20251023 20:15:37.366077 25464 consensus_meta.cc:287] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.107s user 0.003s sys 0.000s
I20251023 20:15:37.366138 23389 raft_consensus.cc:2468] T ea9be2a866cf46cfbec8db23a6d3f572 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.366204 25464 raft_consensus.cc:515] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.366789 25464 leader_election.cc:290] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.368083 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 173) took 8588 ms. Trace:
I20251023 20:15:37.368196 24780 rpcz_store.cc:276] 1023 20:15:28.779628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.779666 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:36.784264 (+8004598us) ts_tablet_manager.cc:1151] Deleting tablet d62dbf044acd4459b26d2e7e50f6206e
1023 20:15:36.784276 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.960336 (+176060us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.963366 (+ 3030us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.368059 (+404693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8004437}
W20251023 20:15:37.368602 23399 consensus_meta.cc:287] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.543s user 0.000s sys 0.000s
I20251023 20:15:37.368724 23399 raft_consensus.cc:2468] T dae906f257c346d7a1b28c9052d600b5 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.368621 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 174) took 8588 ms. Trace:
I20251023 20:15:37.369045 24780 rpcz_store.cc:276] 1023 20:15:28.779946 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.779986 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:37.368473 (+8588487us) ts_tablet_manager.cc:1151] Deleting tablet e5368813164949ff80e27bf59dcc931a
1023 20:15:37.368483 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.368583 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8588318}
I20251023 20:15:37.369329 24780 tablet_replica.cc:333] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:37.369568 24780 raft_consensus.cc:2243] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:37.373299 23665 raft_consensus.cc:2468] T 8d22aa40e5844d78a8679a3ad1b08da9 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:37.374377 23470 leader_election.cc:343] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 370a790a053446b2b06314c7db242270
I20251023 20:15:37.375679 25464 consensus_queue.cc:1048] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.376125 23790 tablet_bootstrap.cc:492] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.376248 23790 ts_tablet_manager.cc:1403] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.000s
I20251023 20:15:37.378638 25464 consensus_queue.cc:1048] T a744214a24cc4bf28654ffa417ce0d3d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.381260 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "255521d943d34713aaf773b15f9168c6" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.381678 23605 leader_election.cc:343] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.381806 23605 leader_election.cc:304] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.381989 25657 raft_consensus.cc:2749] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.376965 23790 raft_consensus.cc:359] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.386962 23790 raft_consensus.cc:385] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.387050 23790 raft_consensus.cc:740] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.387274 23790 consensus_queue.cc:260] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.388851 23790 ts_tablet_manager.cc:1434] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251023 20:15:37.389286 23786 tablet_bootstrap.cc:654] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.392554 25725 raft_consensus.cc:493] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.392745 25725 raft_consensus.cc:515] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.393424 25725 leader_election.cc:290] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.393940 24780 raft_consensus.cc:2272] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:37.398456 24780 ts_tablet_manager.cc:1916] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:37.398533 23333 leader_election.cc:304] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.400702 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e2fec8f4e14036af714427481f2f36" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.400893 23527 raft_consensus.cc:3060] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.401628 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e9e2fec8f4e14036af714427481f2f36" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.401798 23666 raft_consensus.cc:3060] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.408547 25726 raft_consensus.cc:2804] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.408982 25726 raft_consensus.cc:697] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:37.409327 25726 consensus_queue.cc:237] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.398653 23790 tablet_bootstrap.cc:492] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.411860 23790 tablet_bootstrap.cc:654] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.416754 23524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a6a3425a49403e99c8ade114166e46" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.416961 23524 raft_consensus.cc:2468] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.417704 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a6a3425a49403e99c8ade114166e46" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.417874 23665 raft_consensus.cc:2468] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.418583 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3befdfc85bca42bb816e11d1dd832c8c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.419327 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "370a790a053446b2b06314c7db242270" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.419466 23387 raft_consensus.cc:3060] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:37.421377 23471 leader_election.cc:343] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.421967 23333 leader_election.cc:304] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.422441 25594 raft_consensus.cc:2804] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.422516 25594 raft_consensus.cc:493] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.422595 25594 raft_consensus.cc:3060] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.426347 25599 consensus_queue.cc:1048] T 53f33020319d48c791e1156c67b55be6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.432140 23786 tablet_bootstrap.cc:492] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.432315 23786 ts_tablet_manager.cc:1403] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20251023 20:15:37.441625 23786 raft_consensus.cc:359] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.441852 23786 raft_consensus.cc:385] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.443636 23790 tablet_bootstrap.cc:492] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.443776 23790 ts_tablet_manager.cc:1403] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20251023 20:15:37.441919 23786 raft_consensus.cc:740] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.445586 23786 consensus_queue.cc:260] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.445904 23787 tablet_bootstrap.cc:492] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.446058 23787 ts_tablet_manager.cc:1403] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.208s user 0.011s sys 0.013s
I20251023 20:15:37.446169 23786 ts_tablet_manager.cc:1434] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251023 20:15:37.446792 23787 raft_consensus.cc:359] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.446979 23787 raft_consensus.cc:385] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.447031 23787 raft_consensus.cc:740] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.447214 23787 consensus_queue.cc:260] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.447589 23786 tablet_bootstrap.cc:492] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.447659 23787 ts_tablet_manager.cc:1434] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:37.448130 23787 tablet_bootstrap.cc:492] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.449429 23786 tablet_bootstrap.cc:654] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.457201 23790 raft_consensus.cc:359] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.457386 23790 raft_consensus.cc:385] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.457438 23790 raft_consensus.cc:740] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.457666 23790 consensus_queue.cc:260] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.458472 23790 ts_tablet_manager.cc:1434] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251023 20:15:37.458750 23790 tablet_bootstrap.cc:492] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.460173 23790 tablet_bootstrap.cc:654] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.464402 23787 tablet_bootstrap.cc:654] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.467082 23670 raft_consensus.cc:2468] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:37.467832 23470 consensus_peers.cc:597] T 76380b341a304d6ba31648fbd1548f05 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:37.469365 23528 consensus_meta.cc:287] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.165s user 0.001s sys 0.001s
I20251023 20:15:37.469730 23528 raft_consensus.cc:2468] T 82ae1a94b011412c851cd96946744fc4 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.469918 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.127.129:58219 (request call id 829) took 1171 ms. Trace:
I20251023 20:15:37.470034 23528 rpcz_store.cc:276] 1023 20:15:36.298095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.298136 (+ 41us) service_pool.cc:225] Handling call
1023 20:15:37.469880 (+1171744us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:37.470621 23470 leader_election.cc:304] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:37.471135 25595 raft_consensus.cc:515] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.471647 25568 raft_consensus.cc:2804] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.471740 25595 leader_election.cc:290] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.471801 25568 raft_consensus.cc:697] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.472054 25568 consensus_queue.cc:237] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.472290 25282 ts_tablet_manager.cc:1929] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:37.472378 25282 log.cc:1199] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/76380b341a304d6ba31648fbd1548f05
I20251023 20:15:37.472687 25282 ts_tablet_manager.cc:1950] T 76380b341a304d6ba31648fbd1548f05 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:37.476778 23396 raft_consensus.cc:2393] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.486963 24780 ts_tablet_manager.cc:1929] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:37.487100 24780 log.cc:1199] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/252b5480222f49439052c86a4115cc9b
I20251023 20:15:37.487411 24780 ts_tablet_manager.cc:1950] T 252b5480222f49439052c86a4115cc9b P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:37.494715 23787 tablet_bootstrap.cc:492] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.494875 23787 ts_tablet_manager.cc:1403] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20251023 20:15:37.495609 25657 raft_consensus.cc:493] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.495690 23787 raft_consensus.cc:359] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.495743 25657 raft_consensus.cc:515] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.495873 23787 raft_consensus.cc:385] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.495920 23787 raft_consensus.cc:740] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.496120 23787 consensus_queue.cc:260] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.496284 25657 leader_election.cc:290] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.496595 23787 ts_tablet_manager.cc:1434] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:37.496899 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbab01ecb27a426188689408079eaa01" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.497772 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbab01ecb27a426188689408079eaa01" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.498471 23604 leader_election.cc:343] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:37.498849 23605 leader_election.cc:343] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.498942 23605 leader_election.cc:304] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.499298 25276 raft_consensus.cc:2749] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.500713 23787 tablet_bootstrap.cc:492] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.502390 23787 tablet_bootstrap.cc:654] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.521939 23787 tablet_bootstrap.cc:492] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.522099 23787 ts_tablet_manager.cc:1403] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251023 20:15:37.523283 23787 raft_consensus.cc:359] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.523442 23787 raft_consensus.cc:385] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.523488 23787 raft_consensus.cc:740] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.523721 23787 consensus_queue.cc:260] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.524204 23787 ts_tablet_manager.cc:1434] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:37.524464 23787 tablet_bootstrap.cc:492] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.524914 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.525067 23665 raft_consensus.cc:3060] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.524968 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.526420 25599 raft_consensus.cc:493] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.526566 25599 raft_consensus.cc:515] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.527215 25599 leader_election.cc:290] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.528111 23470 leader_election.cc:343] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 9a6a2d2b25824dbdafaf2a9cabb0f061
I20251023 20:15:37.529835 25568 consensus_queue.cc:1048] T 473324b22da14618a84aa1136d081993 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.530696 25595 consensus_queue.cc:1048] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.530903 25724 consensus_queue.cc:1048] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.531967 23787 tablet_bootstrap.cc:654] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.545358 23786 tablet_bootstrap.cc:492] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.545578 23786 ts_tablet_manager.cc:1403] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
I20251023 20:15:37.546428 23786 raft_consensus.cc:359] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.546658 23786 raft_consensus.cc:385] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.546746 23786 raft_consensus.cc:740] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.548101 23790 tablet_bootstrap.cc:492] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.548074 25749 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 690ffde75cd745b6a87925967d54471a)
I20251023 20:15:37.548250 23790 ts_tablet_manager.cc:1403] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.000s
I20251023 20:15:37.548439 25749 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:37.549270 23790 raft_consensus.cc:359] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.549435 23790 raft_consensus.cc:385] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.549489 23790 raft_consensus.cc:740] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.549677 23790 consensus_queue.cc:260] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.549755 25749 leader_election.cc:290] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:37.550084 23787 tablet_bootstrap.cc:492] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.550220 23787 ts_tablet_manager.cc:1403] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20251023 20:15:37.550240 23786 consensus_queue.cc:260] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.550923 23786 ts_tablet_manager.cc:1434] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:37.550889 23787 raft_consensus.cc:359] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.551052 23787 raft_consensus.cc:385] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.551107 23787 raft_consensus.cc:740] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.551738 25748 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 690ffde75cd745b6a87925967d54471a)
I20251023 20:15:37.552338 24925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 807 } } ignore_live_leader: false dest_uuid: "ae5a310b4a514c0f812bda133c756062" is_pre_election: true
I20251023 20:15:37.557363 25748 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:37.558543 24925 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6545140d94b7485aa81c51ad233355cd in term 2.
I20251023 20:15:37.559324 23949 leader_election.cc:304] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd, ae5a310b4a514c0f812bda133c756062; no voters:
I20251023 20:15:37.559695 25749 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251023 20:15:37.559840 25749 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 690ffde75cd745b6a87925967d54471a)
I20251023 20:15:37.559945 25749 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 2 FOLLOWER]: Advancing to term 3
I20251023 20:15:37.560500 24013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 3 candidate_status { last_received { term: 2 index: 807 } } ignore_live_leader: false dest_uuid: "6545140d94b7485aa81c51ad233355cd" is_pre_election: true
I20251023 20:15:37.561183 23787 consensus_queue.cc:260] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.562156 25748 leader_election.cc:290] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:37.563071 23787 ts_tablet_manager.cc:1434] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251023 20:15:37.564769 23787 tablet_bootstrap.cc:492] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.566337 23787 tablet_bootstrap.cc:654] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.566393 25749 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:37.567092 25749 leader_election.cc:290] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 3 election: Requested vote from peers ae5a310b4a514c0f812bda133c756062 (127.22.127.189:33509), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:37.567397 24013 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae5a310b4a514c0f812bda133c756062 in current term 3: Already voted for candidate 6545140d94b7485aa81c51ad233355cd in this term.
I20251023 20:15:37.568802 24925 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 807 } } ignore_live_leader: false dest_uuid: "ae5a310b4a514c0f812bda133c756062"
I20251023 20:15:37.569000 24925 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 2 FOLLOWER]: Advancing to term 3
I20251023 20:15:37.571409 25724 raft_consensus.cc:493] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.571548 25724 raft_consensus.cc:515] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:37.571849 23530 consensus_meta.cc:287] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.146s user 0.001s sys 0.001s
I20251023 20:15:37.571951 23530 raft_consensus.cc:2468] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.572096 23530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.127.129:58219 (request call id 837) took 1147 ms. Trace:
I20251023 20:15:37.572106 25724 leader_election.cc:290] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.572165 23530 rpcz_store.cc:276] 1023 20:15:36.424379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.425584 (+ 1205us) service_pool.cc:225] Handling call
1023 20:15:37.572053 (+1146469us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 20:15:37.572463 25489 consensus_meta.cc:287] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.565s user 0.000s sys 0.002s
I20251023 20:15:37.572547 25489 raft_consensus.cc:515] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.573041 25489 leader_election.cc:290] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.573329 23666 raft_consensus.cc:2468] T e9e2fec8f4e14036af714427481f2f36 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.573441 25739 raft_consensus.cc:493] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.573635 25739 raft_consensus.cc:515] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.574340 25739 leader_election.cc:290] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.579140 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.579345 23666 raft_consensus.cc:2468] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.579929 23470 leader_election.cc:304] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.580783 25739 raft_consensus.cc:2804] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.580952 25739 raft_consensus.cc:493] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.581029 25739 raft_consensus.cc:3060] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.584821 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.585172 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a6a2d2b25824dbdafaf2a9cabb0f061" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.585397 23396 raft_consensus.cc:2468] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.585677 23790 ts_tablet_manager.cc:1434] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20251023 20:15:37.586172 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 378) took 4055 ms. Trace:
I20251023 20:15:37.586253 25282 rpcz_store.cc:276] 1023 20:15:33.530320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.530380 (+ 60us) service_pool.cc:225] Handling call
1023 20:15:37.269929 (+3739549us) ts_tablet_manager.cc:1151] Deleting tablet 76380b341a304d6ba31648fbd1548f05
1023 20:15:37.269938 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.372871 (+102933us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.472184 (+ 99313us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.586152 (+113968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3738153}
I20251023 20:15:37.586651 25282 tablet_replica.cc:333] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:37.586838 25282 raft_consensus.cc:2243] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:37.586964 25282 raft_consensus.cc:2272] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:37.587335 23786 tablet_bootstrap.cc:492] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.587990 25282 ts_tablet_manager.cc:1916] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:37.588119 23787 tablet_bootstrap.cc:492] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.588217 23787 ts_tablet_manager.cc:1403] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251023 20:15:37.588793 23786 tablet_bootstrap.cc:654] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.589031 23787 raft_consensus.cc:359] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.589200 23787 raft_consensus.cc:385] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.589246 23787 raft_consensus.cc:740] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.589425 23787 consensus_queue.cc:260] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.589890 23787 ts_tablet_manager.cc:1434] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:37.590139 23787 tablet_bootstrap.cc:492] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.591777 23787 tablet_bootstrap.cc:654] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.594641 23790 tablet_bootstrap.cc:492] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.595644 23333 leader_election.cc:304] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.595881 25489 raft_consensus.cc:2804] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.595994 25489 raft_consensus.cc:697] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:37.596210 25489 consensus_queue.cc:237] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.597802 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1822fdf1f5f44dd4a742aa3c2f8443cf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.597980 23525 raft_consensus.cc:2468] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.599536 25276 raft_consensus.cc:493] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.599677 25276 raft_consensus.cc:515] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.600205 25276 leader_election.cc:290] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.601504 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3150a8478c144d285e999d5c9270fe0" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.601679 23666 raft_consensus.cc:3060] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:37.601917 23471 leader_election.cc:343] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.603178 23530 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e735945647148478201eeffa1d03632" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.603267 23787 tablet_bootstrap.cc:492] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.603372 23787 ts_tablet_manager.cc:1403] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
W20251023 20:15:37.603729 23604 leader_election.cc:343] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.603871 23471 leader_election.cc:304] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:37.604127 25599 raft_consensus.cc:2804] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.604197 25599 raft_consensus.cc:493] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.604254 25599 raft_consensus.cc:3060] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.604178 23787 raft_consensus.cc:359] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.604305 23787 raft_consensus.cc:385] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.604344 23787 raft_consensus.cc:740] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.604525 23787 consensus_queue.cc:260] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.605129 23787 ts_tablet_manager.cc:1434] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:37.605149 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3150a8478c144d285e999d5c9270fe0" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.605294 23525 raft_consensus.cc:3060] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.605401 23787 tablet_bootstrap.cc:492] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.606940 23787 tablet_bootstrap.cc:654] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.607796 23790 tablet_bootstrap.cc:654] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.610010 23333 leader_election.cc:304] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:37.611338 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e735945647148478201eeffa1d03632" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.612036 23605 leader_election.cc:343] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.612152 23605 leader_election.cc:304] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.612366 25276 raft_consensus.cc:2749] T 2e735945647148478201eeffa1d03632 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.617585 25724 raft_consensus.cc:2804] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.617733 25724 raft_consensus.cc:697] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:37.618167 25724 consensus_queue.cc:237] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.620172 23333 leader_election.cc:304] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.620503 25724 raft_consensus.cc:2804] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.620615 25724 raft_consensus.cc:493] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.620692 25724 raft_consensus.cc:3060] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.622874 25740 consensus_queue.cc:1048] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.626304 25489 raft_consensus.cc:493] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.626430 25489 raft_consensus.cc:515] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.626976 25489 leader_election.cc:290] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.628965 23787 tablet_bootstrap.cc:492] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.629113 23787 ts_tablet_manager.cc:1403] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20251023 20:15:37.629784 23787 raft_consensus.cc:359] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.629946 23787 raft_consensus.cc:385] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.629998 23787 raft_consensus.cc:740] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.630198 23787 consensus_queue.cc:260] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.630661 23787 ts_tablet_manager.cc:1434] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:37.630906 23787 tablet_bootstrap.cc:492] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.632440 23787 tablet_bootstrap.cc:654] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.636552 23786 tablet_bootstrap.cc:492] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.636737 23786 ts_tablet_manager.cc:1403] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20251023 20:15:37.637643 23786 raft_consensus.cc:359] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.637879 23786 raft_consensus.cc:385] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.637964 23786 raft_consensus.cc:740] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.638229 23786 consensus_queue.cc:260] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.638886 23786 ts_tablet_manager.cc:1434] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:37.639195 23786 tablet_bootstrap.cc:492] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.640544 23786 tablet_bootstrap.cc:654] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.646078 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.646279 23662 raft_consensus.cc:2468] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
W20251023 20:15:37.647497 23333 leader_election.cc:343] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1822fdf1f5f44dd4a742aa3c2f8443cf
I20251023 20:15:37.648941 23333 leader_election.cc:304] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.649277 25489 raft_consensus.cc:2804] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.649390 25489 raft_consensus.cc:493] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.649498 25489 raft_consensus.cc:3060] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.657320 23790 tablet_bootstrap.cc:492] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.657514 23790 ts_tablet_manager.cc:1403] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20251023 20:15:37.658456 23790 raft_consensus.cc:359] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.658711 23790 raft_consensus.cc:385] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.658797 23790 raft_consensus.cc:740] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.659029 23790 consensus_queue.cc:260] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.659649 23790 ts_tablet_manager.cc:1434] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:37.659950 23790 tablet_bootstrap.cc:492] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:37.662009 23333 consensus_peers.cc:597] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40c659c273c449939d0dd708d39b393f. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:37.645509 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.650523 25740 raft_consensus.cc:493] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.663206 25740 raft_consensus.cc:515] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.663772 25740 leader_election.cc:290] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.664497 23470 leader_election.cc:343] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: f9bd2b109bfa4804a372965e60ba6e85
I20251023 20:15:37.667577 25276 raft_consensus.cc:493] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.667726 25276 raft_consensus.cc:515] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.668228 25276 leader_election.cc:290] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.669150 23387 raft_consensus.cc:2468] T 370a790a053446b2b06314c7db242270 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.669140 25715 consensus_queue.cc:1048] T ea9be2a866cf46cfbec8db23a6d3f572 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.670946 23787 tablet_bootstrap.cc:492] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.671022 23790 tablet_bootstrap.cc:654] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.671151 23787 ts_tablet_manager.cc:1403] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20251023 20:15:37.671674 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9bd2b109bfa4804a372965e60ba6e85" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.671834 23387 raft_consensus.cc:2468] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.671818 23471 leader_election.cc:304] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:37.672149 25715 raft_consensus.cc:2804] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.672252 25715 raft_consensus.cc:697] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.672466 25715 consensus_queue.cc:237] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.673017 23787 raft_consensus.cc:359] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.673173 23787 raft_consensus.cc:385] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.673218 23787 raft_consensus.cc:740] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.673374 23787 consensus_queue.cc:260] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.673779 23787 ts_tablet_manager.cc:1434] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:37.674113 23787 tablet_bootstrap.cc:492] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.675266 24925 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6545140d94b7485aa81c51ad233355cd in term 3.
I20251023 20:15:37.675544 23787 tablet_bootstrap.cc:654] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:37.675701 25600 consensus_meta.cc:287] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.116s user 0.001s sys 0.002s
I20251023 20:15:37.675786 25600 raft_consensus.cc:515] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.676340 25600 leader_election.cc:290] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.676589 23665 raft_consensus.cc:2468] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.684962 23536 raft_consensus.cc:2393] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.687932 23949 leader_election.cc:304] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6545140d94b7485aa81c51ad233355cd, ae5a310b4a514c0f812bda133c756062; no voters:
I20251023 20:15:37.688156 23790 tablet_bootstrap.cc:492] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.688272 23790 ts_tablet_manager.cc:1403] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20251023 20:15:37.688303 25749 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 3 FOLLOWER]: Leader election won for term 3
I20251023 20:15:37.688761 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.689393 23790 raft_consensus.cc:359] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.689721 23790 raft_consensus.cc:385] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.689786 23790 raft_consensus.cc:740] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.689981 23790 consensus_queue.cc:260] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.690228 24297 consensus_queue.cc:1048] T 82ae1a94b011412c851cd96946744fc4 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.690449 23790 ts_tablet_manager.cc:1434] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:37.690696 23790 tablet_bootstrap.cc:492] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.692085 23790 tablet_bootstrap.cc:654] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.693802 25657 raft_consensus.cc:493] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.693995 25657 raft_consensus.cc:515] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.694623 25657 leader_election.cc:290] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.696009 23787 tablet_bootstrap.cc:492] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.696113 23787 ts_tablet_manager.cc:1403] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20251023 20:15:37.696987 23787 raft_consensus.cc:359] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.697263 25600 raft_consensus.cc:493] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.697381 25600 raft_consensus.cc:515] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.697588 23787 raft_consensus.cc:385] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.697664 23787 raft_consensus.cc:740] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.697881 25600 leader_election.cc:290] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.697834 23787 consensus_queue.cc:260] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.698079 23444 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:37.698369 23787 ts_tablet_manager.cc:1434] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:37.698606 23787 tablet_bootstrap.cc:492] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.699426 25749 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [term 3 LEADER]: Becoming Leader. State: Replica: 6545140d94b7485aa81c51ad233355cd, State: Running, Role: LEADER
I20251023 20:15:37.700023 25749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 807, Committed index: 807, Last appended: 2.807, Last appended by leader: 807, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:37.700765 23787 tablet_bootstrap.cc:654] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.701501 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22b7598fe6734d9098fcaa0a0f4bdc7b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.702134 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "660d84eb63a4490886aba1777d449ee2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.703447 23604 leader_election.cc:343] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.707785 23790 tablet_bootstrap.cc:492] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.707918 23790 ts_tablet_manager.cc:1403] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20251023 20:15:37.708505 23790 raft_consensus.cc:359] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.708675 23790 raft_consensus.cc:385] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.708715 23790 raft_consensus.cc:740] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.708760 25600 raft_consensus.cc:493] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.708885 25600 raft_consensus.cc:515] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.709390 25600 leader_election.cc:290] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.709988 25770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6545140d94b7485aa81c51ad233355cd. Latest consensus state: current_term: 3 leader_uuid: "6545140d94b7485aa81c51ad233355cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:37.710426 25770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:37.712059 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "660d84eb63a4490886aba1777d449ee2" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:37.712563 23604 leader_election.cc:343] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.713763 23790 consensus_queue.cc:260] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.714685 23471 leader_election.cc:304] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:37.714931 25600 raft_consensus.cc:2804] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.715008 25600 raft_consensus.cc:493] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.715058 25600 raft_consensus.cc:3060] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:37.716424 23605 leader_election.cc:343] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.716537 23605 leader_election.cc:304] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.716639 23790 ts_tablet_manager.cc:1434] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251023 20:15:37.716928 23790 tablet_bootstrap.cc:492] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.717317 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.717517 23670 raft_consensus.cc:2468] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.717334 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59598ea40b9e4e5da0ec36f35f1094d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.717833 23662 raft_consensus.cc:2468] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.718520 25772 raft_consensus.cc:493] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.718539 25276 raft_consensus.cc:2749] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.718657 25772 raft_consensus.cc:515] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.718786 25276 raft_consensus.cc:493] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.718815 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22b7598fe6734d9098fcaa0a0f4bdc7b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.718885 25276 raft_consensus.cc:515] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.719154 25772 leader_election.cc:290] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.719280 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.719408 23665 raft_consensus.cc:3060] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.719501 25276 leader_election.cc:290] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:37.720300 23605 leader_election.cc:343] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.720394 23605 leader_election.cc:304] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.720738 25657 raft_consensus.cc:2749] T 22b7598fe6734d9098fcaa0a0f4bdc7b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.721307 23470 leader_election.cc:304] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.721553 23470 leader_election.cc:304] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.721553 25740 raft_consensus.cc:2804] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.721699 25740 raft_consensus.cc:493] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.721760 25740 raft_consensus.cc:3060] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.721762 25741 raft_consensus.cc:2804] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.722007 25741 raft_consensus.cc:493] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.722059 25741 raft_consensus.cc:3060] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.722462 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3df1fd83e3c4ac29a7846736701044a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:37.722942 23605 leader_election.cc:343] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 3a74b3f22b854c1ab2b6cc817b84cf47
W20251023 20:15:37.723095 23605 leader_election.cc:343] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.723686 23786 tablet_bootstrap.cc:492] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.723814 23786 ts_tablet_manager.cc:1403] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.003s
I20251023 20:15:37.727946 23786 raft_consensus.cc:359] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.728101 23786 raft_consensus.cc:385] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.728144 23786 raft_consensus.cc:740] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.713951 23333 leader_election.cc:304] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.729167 25595 raft_consensus.cc:2804] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.729281 25595 raft_consensus.cc:697] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:37.729467 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.729512 25595 consensus_queue.cc:237] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.729730 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59598ea40b9e4e5da0ec36f35f1094d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.730163 23787 tablet_bootstrap.cc:492] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.730284 23787 ts_tablet_manager.cc:1403] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.002s
I20251023 20:15:37.731635 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3df1fd83e3c4ac29a7846736701044a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.732359 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a74b3f22b854c1ab2b6cc817b84cf47" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:37.732962 23604 leader_election.cc:343] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.733158 25775 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251023 20:15:37.733142 23604 leader_election.cc:304] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:37.733493 23604 leader_election.cc:343] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.733798 23790 tablet_bootstrap.cc:654] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:37.735867 23471 leader_election.cc:343] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.736312 23787 raft_consensus.cc:359] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.736460 23787 raft_consensus.cc:385] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.736510 23787 raft_consensus.cc:740] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.736690 23787 consensus_queue.cc:260] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.736935 25657 raft_consensus.cc:2749] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.737213 23787 ts_tablet_manager.cc:1434] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20251023 20:15:37.738517 25568 consensus_queue.cc:1048] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.750887 23787 tablet_bootstrap.cc:492] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.752353 23787 tablet_bootstrap.cc:654] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.756855 23786 consensus_queue.cc:260] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.757367 23786 ts_tablet_manager.cc:1434] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20251023 20:15:37.757728 23786 tablet_bootstrap.cc:492] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.759107 23786 tablet_bootstrap.cc:654] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.765854 25715 raft_consensus.cc:493] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.766005 25715 raft_consensus.cc:515] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.766650 25715 leader_election.cc:290] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.767134 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bce63866d524067b87145b07347ffcf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.767326 23667 raft_consensus.cc:2468] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.767705 23470 leader_election.cc:304] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.768261 25715 raft_consensus.cc:2804] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.768348 25715 raft_consensus.cc:493] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.768412 25715 raft_consensus.cc:3060] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.768803 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bce63866d524067b87145b07347ffcf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.769323 23790 tablet_bootstrap.cc:492] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.769457 23790 ts_tablet_manager.cc:1403] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
W20251023 20:15:37.770143 23471 leader_election.cc:343] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:37.774477 23790 raft_consensus.cc:359] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.774636 23790 raft_consensus.cc:385] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.774679 23790 raft_consensus.cc:740] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.774991 25723 consensus_queue.cc:1048] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.775135 25282 ts_tablet_manager.cc:1929] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:37.775239 25282 log.cc:1199] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/8317a54a5f7b4afe8da9d336268efe43
I20251023 20:15:37.775161 23790 consensus_queue.cc:260] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.775467 25594 raft_consensus.cc:515] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.775913 23790 ts_tablet_manager.cc:1434] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251023 20:15:37.776005 25594 leader_election.cc:290] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.776175 23790 tablet_bootstrap.cc:492] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
W20251023 20:15:37.776243 23532 consensus_meta.cc:287] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.161s user 0.002s sys 0.001s
I20251023 20:15:37.776325 23532 raft_consensus.cc:2468] T 5161336598884742ad403fc88c9d9f0b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.776443 23532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.127.129:58219 (request call id 854) took 1161 ms. Trace:
I20251023 20:15:37.776499 23532 rpcz_store.cc:276] 1023 20:15:36.615019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.615072 (+ 53us) service_pool.cc:225] Handling call
1023 20:15:37.776428 (+1161356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19456}
I20251023 20:15:37.776863 23666 raft_consensus.cc:2468] T b3150a8478c144d285e999d5c9270fe0 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:37.777820 23362 consensus_meta.cc:287] T cde03512dfe24dbdb2fb274affd558cf P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.736s user 0.002s sys 0.000s
W20251023 20:15:37.780350 23366 consensus_meta.cc:287] T 866f7e4cbed341b7b75182891c034c28 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.738s user 0.001s sys 0.000s
I20251023 20:15:37.782461 23532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a6a3425a49403e99c8ade114166e46" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.782634 23532 raft_consensus.cc:3060] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.783277 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98a6a3425a49403e99c8ade114166e46" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.783428 23667 raft_consensus.cc:3060] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.783638 23333 leader_election.cc:304] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.783866 25723 raft_consensus.cc:2804] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.783987 25723 raft_consensus.cc:697] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:37.784214 25723 consensus_queue.cc:237] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.784893 23790 tablet_bootstrap.cc:654] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.785687 25466 raft_consensus.cc:493] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.785816 25466 raft_consensus.cc:515] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.786356 25466 leader_election.cc:290] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.787374 23470 leader_election.cc:343] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 857da3642f0846ce92aaed0491e89644
I20251023 20:15:37.791101 23786 tablet_bootstrap.cc:492] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.791224 23786 ts_tablet_manager.cc:1403] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.002s
I20251023 20:15:37.793533 25282 ts_tablet_manager.cc:1950] T 8317a54a5f7b4afe8da9d336268efe43 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:37.791857 23786 raft_consensus.cc:359] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.794585 23786 raft_consensus.cc:385] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.794646 23786 raft_consensus.cc:740] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.794836 23786 consensus_queue.cc:260] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.795284 23786 ts_tablet_manager.cc:1434] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:37.795531 23786 tablet_bootstrap.cc:492] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:37.796963 23786 tablet_bootstrap.cc:654] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.804723 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_baf578c1b5e748e09f244f10685992a8 [id=006294844eac4bcabdd6508738bdc4cf]
I20251023 20:15:37.809011 25568 raft_consensus.cc:493] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.809206 25568 raft_consensus.cc:515] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.809345 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "857da3642f0846ce92aaed0491e89644" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.809500 23389 raft_consensus.cc:2468] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.809834 25568 leader_election.cc:290] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.811097 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d227b45bfcc64d76a51cff347e6e1486" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.811124 23471 leader_election.cc:304] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:37.811261 23662 raft_consensus.cc:2468] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.811929 23470 leader_election.cc:304] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.812399 25568 raft_consensus.cc:2804] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.812475 25568 raft_consensus.cc:493] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.812475 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d227b45bfcc64d76a51cff347e6e1486" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.812534 25568 raft_consensus.cc:3060] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.812690 23389 raft_consensus.cc:2468] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:37.813021 25466 consensus_queue.cc:1048] T e658aa360e524a9f9bdb35c1f68ae426 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:37.813423 25466 raft_consensus.cc:2804] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.813488 25466 raft_consensus.cc:493] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.813534 25466 raft_consensus.cc:3060] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.815141 23787 tablet_bootstrap.cc:492] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.815284 23787 ts_tablet_manager.cc:1403] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.000s
I20251023 20:15:37.815747 25776 raft_consensus.cc:493] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.816951 23787 raft_consensus.cc:359] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.817719 23787 raft_consensus.cc:385] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.817788 23787 raft_consensus.cc:740] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.817968 23787 consensus_queue.cc:260] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.818502 23787 ts_tablet_manager.cc:1434] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:37.818801 23787 tablet_bootstrap.cc:492] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:37.818955 25594 raft_consensus.cc:493] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.818989 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_4d8885fad7e54f0eae2d0720f3062ec0 [id=0357e034941f45d2b977a566dfee9346]
I20251023 20:15:37.819092 25594 raft_consensus.cc:515] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.819664 25594 leader_election.cc:290] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.819736 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_05604aaaf6604c4c8f929ac257e5824e [id=04588122a9924ac5bcef60dedde8b1cd]
I20251023 20:15:37.820123 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_4c838e5585b948b9ba710086a76a2f3f [id=04d988798f7f4b56be2fe5ed680539d4]
W20251023 20:15:37.820510 23333 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:37.820776 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_30b62bd8bf6147a7a5800584576db269 [id=054cce203d7e4437a8a7aa22df487453]
W20251023 20:15:37.821360 23333 leader_election.cc:343] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:37.821858 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3]
I20251023 20:15:37.821818 23333 leader_election.cc:304] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.822031 23787 tablet_bootstrap.cc:654] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.822062 25595 raft_consensus.cc:2749] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:37.826642 23790 tablet_bootstrap.cc:492] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.826783 23790 ts_tablet_manager.cc:1403] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20251023 20:15:37.827474 23790 raft_consensus.cc:359] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.827656 23790 raft_consensus.cc:385] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.827708 23790 raft_consensus.cc:740] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.827939 23790 consensus_queue.cc:260] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.829808 23790 ts_tablet_manager.cc:1434] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:37.830056 23790 tablet_bootstrap.cc:492] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.832041 23790 tablet_bootstrap.cc:654] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.832913 25723 raft_consensus.cc:493] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.833036 25723 raft_consensus.cc:515] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.833406 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3c3fd30fc677490da11231e62dd177aa [id=09f8335a5b03409c90559c8313f5d135]
I20251023 20:15:37.833645 25723 leader_election.cc:290] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.834187 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.834391 23662 raft_consensus.cc:2468] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.834753 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e851810007344c51aa7a54934d6ad579 [id=0ddefd3e67364bc8bf23971811893894]
I20251023 20:15:37.835299 23333 leader_election.cc:304] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.835528 25723 raft_consensus.cc:2804] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.835099 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.835611 25723 raft_consensus.cc:493] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.835669 25723 raft_consensus.cc:3060] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.845890 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113]
I20251023 20:15:37.849628 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_50879e22af484b719900c9b120534de0 [id=1245153d2d05472a9f20bc69214ca3ce]
I20251023 20:15:37.850421 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_8cda343a0b0c499b9d4190bb861443a7 [id=124ec9a7c9574e54b885ca5b31aec53f]
I20251023 20:15:37.815876 25776 raft_consensus.cc:515] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.851745 25776 leader_election.cc:290] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.854375 23604 leader_election.cc:304] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:37.855304 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_511899f307ee49fb8612942d4b966ee2 [id=12648eb0bb8b4909814e3f2cb9cec1ae]
I20251023 20:15:37.856706 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a45d9b3031b46d390e0a1010489c718" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:37.857338 23790 tablet_bootstrap.cc:492] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.857522 23790 ts_tablet_manager.cc:1403] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.007s
I20251023 20:15:37.858326 23790 raft_consensus.cc:359] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.858552 23790 raft_consensus.cc:385] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.858650 23790 raft_consensus.cc:740] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.858878 23790 consensus_queue.cc:260] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.859385 23790 ts_tablet_manager.cc:1434] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:37.859700 23790 tablet_bootstrap.cc:492] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.861742 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28]
I20251023 20:15:37.862035 23790 tablet_bootstrap.cc:654] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.863233 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_5a52905cde274ed386bff112ef98397a [id=1674ccf219e44d6e84ad9f62755ff1e5]
I20251023 20:15:37.863899 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e6fec6ab98474529a2fdf84e3d3cdd2d [id=183b38fa40b949b8b4ca994b1a5efdd1]
I20251023 20:15:37.866253 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73]
I20251023 20:15:37.866338 25595 raft_consensus.cc:493] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:37.866474 25595 raft_consensus.cc:515] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.867038 25595 leader_election.cc:290] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.869815 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2c022d78cac4959984d608d64df851b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:37.870007 23536 raft_consensus.cc:2468] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.870832 23333 leader_election.cc:304] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.870924 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_1b20af8a893a4715affda818cffeca13 [id=201b60a519df46fcb92de45acff546cc]
I20251023 20:15:37.871035 25725 raft_consensus.cc:2804] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:37.871107 25725 raft_consensus.cc:493] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:37.871163 25725 raft_consensus.cc:3060] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.871315 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_869b3f73ab4446369b8ab49294b2d32d [id=20d0cb227c7242b29c5ee5423268da12]
W20251023 20:15:37.871677 25262 consensus_meta.cc:287] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.212s user 0.001s sys 0.001s
I20251023 20:15:37.871774 25262 raft_consensus.cc:515] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.871974 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3a5e9b90e86340ed80fdb4a8679777e6 [id=22d88af65ebd43389f6e58573143816c]
I20251023 20:15:37.872359 25262 leader_election.cc:290] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.872377 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_25427c3d36fb4ba4a01ac362787c81b8 [id=248b68f25e64412ba27e4c74b27c59d8]
W20251023 20:15:37.872725 25235 consensus_meta.cc:287] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.856s user 0.002s sys 0.001s
I20251023 20:15:37.872848 25235 raft_consensus.cc:515] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.873410 25235 leader_election.cc:290] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.874075 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 403) took 2572 ms. Trace:
I20251023 20:15:37.874141 25492 rpcz_store.cc:276] 1023 20:15:35.301623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.301660 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:36.472176 (+1170516us) ts_tablet_manager.cc:1151] Deleting tablet c9bfbf0acdc84f2f8fe553694ee9897c
1023 20:15:36.472216 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:36.561257 (+ 89041us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:36.659474 (+ 98217us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.874062 (+1214588us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2525,"tablet-delete.queue_time_us":1170354}
I20251023 20:15:37.874485 25492 tablet_replica.cc:333] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:37.874655 25492 raft_consensus.cc:2243] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:37.874783 25492 raft_consensus.cc:2272] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:37.875291 25492 ts_tablet_manager.cc:1916] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:37.876894 23790 tablet_bootstrap.cc:492] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:37.877013 23790 ts_tablet_manager.cc:1403] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20251023 20:15:37.877380 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 175) took 9097 ms. Trace:
I20251023 20:15:37.877450 24780 rpcz_store.cc:276] 1023 20:15:28.780363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.780400 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:37.369266 (+8588866us) ts_tablet_manager.cc:1151] Deleting tablet 252b5480222f49439052c86a4115cc9b
1023 20:15:37.369275 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.467390 (+ 98115us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.486830 (+ 19440us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.877365 (+390535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8588721}
I20251023 20:15:37.877861 23790 raft_consensus.cc:359] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.878000 23790 raft_consensus.cc:385] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.878046 23790 raft_consensus.cc:740] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.878239 23790 consensus_queue.cc:260] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.879048 23790 ts_tablet_manager.cc:1434] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251023 20:15:37.880792 25167 consensus_meta.cc:287] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.036s user 0.000s sys 0.000s
I20251023 20:15:37.880884 25167 raft_consensus.cc:515] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:37.881404 25592 consensus_meta.cc:287] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.626s user 0.002s sys 0.000s
I20251023 20:15:37.881487 25592 raft_consensus.cc:515] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.881871 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8]
I20251023 20:15:37.882546 25592 leader_election.cc:290] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.883138 23531 consensus_meta.cc:287] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.993s user 0.002s sys 0.000s
I20251023 20:15:37.883172 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_5b25a14abbd84fbe84dc7edc07272c09 [id=276b2d82a9454bdc83331ebbcceed582]
I20251023 20:15:37.883213 23531 raft_consensus.cc:2468] T 01fa5f9c700b458ea147373d78551573 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.883342 23531 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.127.129:58219 (request call id 804) took 1993 ms. Trace:
I20251023 20:15:37.883406 23531 rpcz_store.cc:276] 1023 20:15:35.889826 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.890024 (+ 198us) service_pool.cc:225] Handling call
1023 20:15:37.883310 (+1993286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:37.883572 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7908a191ee99400093d9f2deb05e52d2 [id=2ad9425327fb485d89c61e102c983b30]
I20251023 20:15:37.883680 23529 raft_consensus.cc:2393] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
W20251023 20:15:37.884066 25172 consensus_meta.cc:287] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.753s user 0.001s sys 0.001s
I20251023 20:15:37.884138 25172 raft_consensus.cc:515] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.884598 25172 leader_election.cc:290] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.885284 25470 consensus_meta.cc:287] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.925s user 0.002s sys 0.000s
I20251023 20:15:37.885376 25470 raft_consensus.cc:515] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.885929 25470 leader_election.cc:290] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.886274 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d491cd3951344335be8170dcdcb94c4c [id=2c61abbba33e4ed4805127246e3cd0ba]
I20251023 20:15:37.886831 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]
W20251023 20:15:37.887061 23533 consensus_meta.cc:287] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.843s user 0.002s sys 0.000s
I20251023 20:15:37.887130 23533 raft_consensus.cc:2468] T 8d22aa40e5844d78a8679a3ad1b08da9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:37.887470 23398 consensus_meta.cc:287] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.758s user 0.002s sys 0.000s
I20251023 20:15:37.887539 23398 raft_consensus.cc:2468] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.887677 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_8d1521c5f05d4bb5a060ed68c5fa6ca5 [id=2f0aa245600741b3a778c959b60b936b]
W20251023 20:15:37.889871 23497 consensus_meta.cc:287] T 6313bf6e463b42bd9990ec50904a3b60 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.061s user 0.004s sys 0.000s
I20251023 20:15:37.890067 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ec09021e59f8464f81cb6ceda1f1c567 [id=2fd818d424a44efdab6cf6fda7537ee8]
I20251023 20:15:37.890163 23497 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48312 (request call id 468) took 1074 ms. Trace:
I20251023 20:15:37.890237 23497 rpcz_store.cc:276] 1023 20:15:36.815676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.815992 (+ 316us) service_pool.cc:225] Handling call
1023 20:15:36.825201 (+ 9209us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1023 20:15:36.825226 (+ 25us) ts_tablet_manager.cc:682] Creating new metadata...
1023 20:15:36.828477 (+ 3251us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.890005 (+1061528us) tablet_replica.cc:206] Creating consensus instance
1023 20:15:37.890149 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:37.890465 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f]
W20251023 20:15:37.890607 24821 consensus_meta.cc:287] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.714s user 0.003s sys 0.000s
I20251023 20:15:37.890691 24821 raft_consensus.cc:515] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.890852 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_87caa0c69120413a92c6b65337baad14 [id=3227f5fd7d78482c99fb9f654ba8e3c6]
I20251023 20:15:37.891227 24821 leader_election.cc:290] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.891247 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718]
W20251023 20:15:37.893649 23504 consensus_meta.cc:287] T cde03512dfe24dbdb2fb274affd558cf P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.074s user 0.002s sys 0.000s
I20251023 20:15:37.893908 23504 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48312 (request call id 469) took 1078 ms. Trace:
I20251023 20:15:37.893970 23504 rpcz_store.cc:276] 1023 20:15:36.815794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.816328 (+ 534us) service_pool.cc:225] Handling call
1023 20:15:36.816641 (+ 313us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1023 20:15:36.816660 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1023 20:15:36.819456 (+ 2796us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.893770 (+1074314us) tablet_replica.cc:206] Creating consensus instance
1023 20:15:37.893896 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:37.855222 23787 tablet_bootstrap.cc:492] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:37.895205 23787 ts_tablet_manager.cc:1403] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.000s
I20251023 20:15:37.895797 25724 raft_consensus.cc:515] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.896000 23787 raft_consensus.cc:359] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.896165 23787 raft_consensus.cc:385] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251023 20:15:37.896171 23508 consensus_meta.cc:287] T c615066185c6472ba13bae91f72ee67c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.077s user 0.001s sys 0.000s
I20251023 20:15:37.896212 23787 raft_consensus.cc:740] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.896333 25724 leader_election.cc:290] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.896418 23508 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48312 (request call id 466) took 1085 ms. Trace:
I20251023 20:15:37.896488 23508 rpcz_store.cc:276] 1023 20:15:36.811061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.811221 (+ 160us) service_pool.cc:225] Handling call
1023 20:15:36.816364 (+ 5143us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1023 20:15:36.816391 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1023 20:15:36.819459 (+ 3068us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.896282 (+1076823us) tablet_replica.cc:206] Creating consensus instance
1023 20:15:37.896405 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 20:15:37.896608 23372 consensus_meta.cc:287] T ec4817cd994d41eaa6aee3dafab4d35e P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.864s user 0.000s sys 0.003s
W20251023 20:15:37.896910 23375 consensus_meta.cc:287] T 6313bf6e463b42bd9990ec50904a3b60 P d220aba7c9c0400398ffde917376ee48: Time spent flushing consensus metadata: real 0.855s user 0.002s sys 0.000s
W20251023 20:15:37.897185 23499 consensus_meta.cc:287] T 1705a87bfa7e4f7caa00b4666fd0514a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.785s user 0.002s sys 0.001s
I20251023 20:15:37.897421 23499 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48312 (request call id 458) took 1789 ms. Trace:
I20251023 20:15:37.897476 23499 rpcz_store.cc:276] 1023 20:15:36.107943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.108043 (+ 100us) service_pool.cc:225] Handling call
1023 20:15:36.108549 (+ 506us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1023 20:15:36.108583 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1023 20:15:36.112438 (+ 3855us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.897301 (+1784863us) tablet_replica.cc:206] Creating consensus instance
1023 20:15:37.897410 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 20:15:37.897786 25566 consensus_meta.cc:287] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.953s user 0.003s sys 0.000s
I20251023 20:15:37.897861 25566 raft_consensus.cc:515] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.898324 25566 leader_election.cc:290] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.898588 25489 raft_consensus.cc:515] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.899081 25489 leader_election.cc:290] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.899052 25723 raft_consensus.cc:515] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.899127 25725 raft_consensus.cc:515] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.899505 25725 leader_election.cc:290] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.900527 25462 consensus_meta.cc:287] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.807s user 0.002s sys 0.000s
I20251023 20:15:37.900614 25462 raft_consensus.cc:515] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.902436 23391 raft_consensus.cc:2393] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.903970 24780 tablet_replica.cc:333] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:37.904167 24780 raft_consensus.cc:2243] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:37.904325 24780 raft_consensus.cc:2272] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:37.904867 24780 ts_tablet_manager.cc:1916] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:37.905196 23387 raft_consensus.cc:2393] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.899513 25723 leader_election.cc:290] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.908159 25554 consensus_meta.cc:287] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.804s user 0.001s sys 0.001s
I20251023 20:15:37.908243 25554 raft_consensus.cc:515] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.908828 25554 leader_election.cc:290] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.911057 25454 consensus_meta.cc:287] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.798s user 0.000s sys 0.001s
I20251023 20:15:37.911139 25454 raft_consensus.cc:515] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.911609 25454 leader_election.cc:290] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.896430 23787 consensus_queue.cc:260] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.912369 23787 ts_tablet_manager.cc:1434] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20251023 20:15:37.912688 23787 tablet_bootstrap.cc:492] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
W20251023 20:15:37.913395 25596 consensus_meta.cc:287] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.655s user 0.003s sys 0.000s
I20251023 20:15:37.913480 25596 raft_consensus.cc:515] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.913933 25596 leader_election.cc:290] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.915230 25619 consensus_meta.cc:287] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.672s user 0.001s sys 0.000s
I20251023 20:15:37.915309 25619 raft_consensus.cc:515] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.915781 25619 leader_election.cc:290] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:37.916340 23527 consensus_meta.cc:287] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.515s user 0.000s sys 0.000s
I20251023 20:15:37.916409 23527 raft_consensus.cc:2468] T e9e2fec8f4e14036af714427481f2f36 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.917253 23526 raft_consensus.cc:2393] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.918893 25739 raft_consensus.cc:515] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:37.919209 23514 consensus_meta.cc:287] T 866f7e4cbed341b7b75182891c034c28 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.103s user 0.001s sys 0.000s
I20251023 20:15:37.919329 25739 leader_election.cc:290] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.919492 23514 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48312 (request call id 467) took 1108 ms. Trace:
I20251023 20:15:37.919567 23514 rpcz_store.cc:276] 1023 20:15:36.811165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.811217 (+ 52us) service_pool.cc:225] Handling call
1023 20:15:36.812533 (+ 1316us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1023 20:15:36.812566 (+ 33us) ts_tablet_manager.cc:682] Creating new metadata...
1023 20:15:36.815734 (+ 3168us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.919336 (+1103602us) tablet_replica.cc:206] Creating consensus instance
1023 20:15:37.919478 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251023 20:15:37.921370 23534 consensus_meta.cc:287] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.828s user 0.002s sys 0.000s
I20251023 20:15:37.921447 23534 raft_consensus.cc:2468] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc6522db11e4960a9aad84435371a32 in term 1.
W20251023 20:15:37.867304 23605 leader_election.cc:343] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:37.921790 23605 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 54501.3us
I20251023 20:15:37.868479 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2c022d78cac4959984d608d64df851b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:37.923429 23662 raft_consensus.cc:2468] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:37.873708 23665 raft_consensus.cc:2468] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.923988 23528 raft_consensus.cc:2393] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.878456 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 373) took 4350 ms. Trace:
I20251023 20:15:37.924361 25282 rpcz_store.cc:276] 1023 20:15:33.527796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.539774 (+ 11978us) service_pool.cc:225] Handling call
1023 20:15:37.586608 (+4046834us) ts_tablet_manager.cc:1151] Deleting tablet 8317a54a5f7b4afe8da9d336268efe43
1023 20:15:37.586617 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.676860 (+ 90243us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.775018 (+ 98158us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.878434 (+103416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4046637}
I20251023 20:15:37.924717 25282 tablet_replica.cc:333] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:37.924922 25282 raft_consensus.cc:2243] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:37.882871 23667 raft_consensus.cc:2468] T 98a6a3425a49403e99c8ade114166e46 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.926301 23525 raft_consensus.cc:2468] T b3150a8478c144d285e999d5c9270fe0 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
W20251023 20:15:37.926820 25465 consensus_meta.cc:287] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 0.704s user 0.001s sys 0.000s
I20251023 20:15:37.926905 25465 raft_consensus.cc:515] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.927423 25465 leader_election.cc:290] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.928610 25740 raft_consensus.cc:515] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.928675 25600 raft_consensus.cc:515] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.929109 25600 leader_election.cc:290] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.929503 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.929684 23525 raft_consensus.cc:2393] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.929834 23333 leader_election.cc:304] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:37.930637 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2c022d78cac4959984d608d64df851b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.930764 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db758a7a7a89410f8ac1da72048fbc56" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.930923 23528 raft_consensus.cc:3060] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.932015 23529 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1822fdf1f5f44dd4a742aa3c2f8443cf" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.932150 23529 raft_consensus.cc:3060] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.933599 25715 raft_consensus.cc:515] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.933683 23536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.933847 23536 raft_consensus.cc:2393] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.934098 25715 leader_election.cc:290] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.934468 23529 raft_consensus.cc:2468] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.935236 23531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.935422 23531 raft_consensus.cc:2393] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.925061 25282 raft_consensus.cc:2272] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:37.936882 25282 ts_tablet_manager.cc:1916] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:37.937005 23532 raft_consensus.cc:2468] T 98a6a3425a49403e99c8ade114166e46 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.937217 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "225ce6aa52c54f76bd56247f01219dbc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.937383 23669 raft_consensus.cc:3060] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.931922 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2202632a57514c849760509360113078" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.938278 23664 raft_consensus.cc:2393] T 2202632a57514c849760509360113078 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.933225 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a2c022d78cac4959984d608d64df851b" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.938604 23663 raft_consensus.cc:3060] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.939038 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa0a46ffa626447eab4cdb956e798005" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.939200 23391 raft_consensus.cc:3060] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.939909 23669 raft_consensus.cc:2468] T 225ce6aa52c54f76bd56247f01219dbc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.940330 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.940475 23668 raft_consensus.cc:3060] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.941216 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19da816a483a4d5b9dc768ef697d54ec" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.941371 23387 raft_consensus.cc:3060] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.941787 23391 raft_consensus.cc:2468] T fa0a46ffa626447eab4cdb956e798005 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.942848 23668 raft_consensus.cc:2468] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.943408 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1efa4fcaeb049b484ca59b12025ce9a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.943554 23398 raft_consensus.cc:3060] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.943885 23387 raft_consensus.cc:2468] T 19da816a483a4d5b9dc768ef697d54ec P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.933284 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.944378 23667 raft_consensus.cc:3060] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.945724 25172 raft_consensus.cc:2804] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.945848 25172 raft_consensus.cc:697] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:37.946026 23398 raft_consensus.cc:2468] T a1efa4fcaeb049b484ca59b12025ce9a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.946132 25172 consensus_queue.cc:237] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.946676 23667 raft_consensus.cc:2468] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.934808 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db758a7a7a89410f8ac1da72048fbc56" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.946959 23662 raft_consensus.cc:3060] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.949255 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.949376 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.949460 23389 raft_consensus.cc:2393] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.949507 23660 raft_consensus.cc:3060] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.949710 23659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa59141039de402e9ccce427604ecb31" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.949781 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.949848 23659 raft_consensus.cc:3060] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.949888 23670 raft_consensus.cc:3060] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.950131 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b446550686024c3ca80f106112a2110f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.950130 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb7935c4cf64e2f802ca12c1f63317b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.950249 23664 raft_consensus.cc:3060] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.950248 23396 raft_consensus.cc:3060] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.950209 23656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bce63866d524067b87145b07347ffcf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.950459 23656 raft_consensus.cc:3060] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.949719 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b446550686024c3ca80f106112a2110f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.950765 23665 raft_consensus.cc:3060] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.951149 23790 tablet_bootstrap.cc:492] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:37.951162 23661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d71f0cfdb6449fbd74b854c15326d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.951287 23661 raft_consensus.cc:3060] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.953038 23790 tablet_bootstrap.cc:654] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.953676 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.954015 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa59141039de402e9ccce427604ecb31" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.954138 23399 raft_consensus.cc:3060] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.955273 23396 raft_consensus.cc:2468] T b446550686024c3ca80f106112a2110f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.955737 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.955919 23392 raft_consensus.cc:2393] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.956063 23333 leader_election.cc:304] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:37.956768 23399 raft_consensus.cc:2468] T aa59141039de402e9ccce427604ecb31 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.957216 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9bd2b109bfa4804a372965e60ba6e85" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.957352 23389 raft_consensus.cc:3060] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.959753 23389 raft_consensus.cc:2468] T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.960150 25592 raft_consensus.cc:2804] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.960259 25592 raft_consensus.cc:697] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:37.960492 25592 consensus_queue.cc:237] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.961719 23787 tablet_bootstrap.cc:654] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:37.962190 23662 raft_consensus.cc:2468] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.962865 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19da816a483a4d5b9dc768ef697d54ec" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.963027 23667 raft_consensus.cc:3060] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.964371 23660 raft_consensus.cc:2468] T 6abfda15e6c242b588d4ed4401008156 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.965391 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb7935c4cf64e2f802ca12c1f63317b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.965548 23670 raft_consensus.cc:2468] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.965603 23387 raft_consensus.cc:3060] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:37.942610 23511 consensus_meta.cc:287] T ec4817cd994d41eaa6aee3dafab4d35e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent flushing consensus metadata: real 1.117s user 0.002s sys 0.000s
I20251023 20:15:37.965968 23511 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48312 (request call id 470) took 1150 ms. Trace:
I20251023 20:15:37.966049 23511 rpcz_store.cc:276] 1023 20:15:36.815939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.819506 (+ 3567us) service_pool.cc:225] Handling call
1023 20:15:36.822168 (+ 2662us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1023 20:15:36.822197 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1023 20:15:36.825346 (+ 3149us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.965789 (+1140443us) tablet_replica.cc:206] Creating consensus instance
1023 20:15:37.965953 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251023 20:15:37.945319 25568 raft_consensus.cc:515] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.948320 23470 leader_election.cc:304] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:37.966634 23663 raft_consensus.cc:2468] T a2c022d78cac4959984d608d64df851b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.966825 25568 leader_election.cc:290] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.966960 25568 raft_consensus.cc:2804] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.967048 25568 raft_consensus.cc:697] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
W20251023 20:15:37.967173 23470 leader_election.cc:343] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: f9bd2b109bfa4804a372965e60ba6e85
I20251023 20:15:37.967247 25568 consensus_queue.cc:237] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.967330 23470 leader_election.cc:304] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.967530 23470 leader_election.cc:304] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.968276 23662 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1efa4fcaeb049b484ca59b12025ce9a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.968443 23662 raft_consensus.cc:3060] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.949136 25466 raft_consensus.cc:515] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.968772 23670 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.968896 23656 raft_consensus.cc:2468] T 4bce63866d524067b87145b07347ffcf P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.968948 23670 raft_consensus.cc:2393] T c3d145a33b964a4ca1f0001ac6cf990b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.954660 25167 leader_election.cc:290] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.969238 25466 leader_election.cc:290] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.969446 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "225ce6aa52c54f76bd56247f01219dbc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.969586 23527 raft_consensus.cc:3060] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.969631 25464 raft_consensus.cc:2804] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.969735 25464 raft_consensus.cc:697] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.969945 25464 consensus_queue.cc:237] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.970757 25566 raft_consensus.cc:2804] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.970772 23662 raft_consensus.cc:2468] T a1efa4fcaeb049b484ca59b12025ce9a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.970870 25566 raft_consensus.cc:697] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.968108 23387 raft_consensus.cc:2468] T 0eb7935c4cf64e2f802ca12c1f63317b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.971077 25566 consensus_queue.cc:237] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.971216 23659 raft_consensus.cc:2468] T aa59141039de402e9ccce427604ecb31 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.972242 23527 raft_consensus.cc:2468] T 225ce6aa52c54f76bd56247f01219dbc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.972573 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa0a46ffa626447eab4cdb956e798005" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.972707 23668 raft_consensus.cc:3060] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.973660 23664 raft_consensus.cc:2468] T 0eb7935c4cf64e2f802ca12c1f63317b P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.976019 23665 raft_consensus.cc:2468] T b446550686024c3ca80f106112a2110f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.976089 23661 raft_consensus.cc:2468] T a4d71f0cfdb6449fbd74b854c15326d6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.977607 23668 raft_consensus.cc:2468] T fa0a46ffa626447eab4cdb956e798005 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.978380 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4d71f0cfdb6449fbd74b854c15326d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.978583 23391 raft_consensus.cc:3060] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.982054 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:37.982254 23398 raft_consensus.cc:2393] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.983649 24902 ts_manager.cc:194] Registered new tserver with Master: ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:37.960246 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:37.961787 25599 raft_consensus.cc:515] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.984776 25599 leader_election.cc:290] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.977859 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf38936bb12846bd81a2c79ffbd18ccb" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.980427 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d227b45bfcc64d76a51cff347e6e1486" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.985757 23669 raft_consensus.cc:2393] T cf38936bb12846bd81a2c79ffbd18ccb P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.980567 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60e8942deb7544a09b21c3d1c2a3f99f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.985879 23663 raft_consensus.cc:2393] T 60e8942deb7544a09b21c3d1c2a3f99f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:37.985728 23660 raft_consensus.cc:3060] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.961985 25462 leader_election.cc:290] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.962654 23528 raft_consensus.cc:2468] T db758a7a7a89410f8ac1da72048fbc56 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:37.930764 23525 raft_consensus.cc:3060] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:37.931192 25741 raft_consensus.cc:515] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.988561 23660 raft_consensus.cc:2468] T d227b45bfcc64d76a51cff347e6e1486 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:37.988724 25741 leader_election.cc:290] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.972402 25740 leader_election.cc:290] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:37.957975 23786 tablet_bootstrap.cc:492] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:37.989148 23786 ts_tablet_manager.cc:1403] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.194s user 0.003s sys 0.006s
I20251023 20:15:37.989897 23786 raft_consensus.cc:359] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.993948 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd548fd9de284377ab557bd98f621068" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:37.994200 23666 raft_consensus.cc:2393] T cd548fd9de284377ab557bd98f621068 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:37.994577 23786 raft_consensus.cc:385] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:37.994694 23786 raft_consensus.cc:740] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:37.994939 23786 consensus_queue.cc:260] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.978669 23470 leader_election.cc:304] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.995362 25740 raft_consensus.cc:2804] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.995462 23470 leader_election.cc:304] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.995668 25462 raft_consensus.cc:2804] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.995704 23470 leader_election.cc:304] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.995785 25462 raft_consensus.cc:697] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.995911 23470 leader_election.cc:304] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.995990 25462 consensus_queue.cc:237] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.996085 25465 raft_consensus.cc:2804] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.996129 23470 leader_election.cc:304] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.996172 25465 raft_consensus.cc:697] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.996311 23470 leader_election.cc:304] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:37.996344 25465 consensus_queue.cc:237] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.995469 25740 raft_consensus.cc:697] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.996759 25740 consensus_queue.cc:237] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.995893 25599 raft_consensus.cc:2804] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.997624 24780 ts_tablet_manager.cc:1929] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:37.997786 25554 raft_consensus.cc:2804] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.997885 25554 raft_consensus.cc:697] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.998067 25554 consensus_queue.cc:237] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.998325 25619 raft_consensus.cc:2804] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:37.998428 25619 raft_consensus.cc:697] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.998644 25619 consensus_queue.cc:237] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:37.998919 24780 log.cc:1199] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/79e3d422a66f432789c0cf4a00e57af2
I20251023 20:15:37.999255 24780 ts_tablet_manager.cc:1950] T 79e3d422a66f432789c0cf4a00e57af2 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:37.999372 25599 raft_consensus.cc:697] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:37.999593 25599 consensus_queue.cc:237] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.001653 23787 tablet_bootstrap.cc:492] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.001752 23667 raft_consensus.cc:2468] T 19da816a483a4d5b9dc768ef697d54ec P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.001785 23787 ts_tablet_manager.cc:1403] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.002s
I20251023 20:15:38.002517 23787 raft_consensus.cc:359] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.002677 23787 raft_consensus.cc:385] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.002724 23787 raft_consensus.cc:740] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.002943 23787 consensus_queue.cc:260] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.003484 23787 ts_tablet_manager.cc:1434] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:38.003757 23787 tablet_bootstrap.cc:492] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.004565 25282 ts_tablet_manager.cc:1929] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:38.004674 25282 log.cc:1199] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/3befdfc85bca42bb816e11d1dd832c8c
I20251023 20:15:38.004922 25282 ts_tablet_manager.cc:1950] T 3befdfc85bca42bb816e11d1dd832c8c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:38.005200 23787 tablet_bootstrap.cc:654] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.009055 25276 raft_consensus.cc:493] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.009193 25276 raft_consensus.cc:515] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.009738 25276 leader_election.cc:290] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:37.990792 23525 raft_consensus.cc:2468] T a2c022d78cac4959984d608d64df851b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:38.010331 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0079496170849c6b3c233e1cd78e40e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.010913 23605 leader_election.cc:343] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.011346 23391 raft_consensus.cc:2468] T a4d71f0cfdb6449fbd74b854c15326d6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.014532 23786 ts_tablet_manager.cc:1434] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20251023 20:15:38.014823 23786 tablet_bootstrap.cc:492] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.016243 23786 tablet_bootstrap.cc:654] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:38.017118 23470 leader_election.cc:343] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 857da3642f0846ce92aaed0491e89644
I20251023 20:15:38.017283 23470 leader_election.cc:304] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.017477 25464 raft_consensus.cc:2804] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.017606 23470 leader_election.cc:304] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.017794 25715 raft_consensus.cc:2804] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.017632 25464 raft_consensus.cc:697] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.017867 23470 leader_election.cc:304] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.018143 25464 consensus_queue.cc:237] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.018296 25466 raft_consensus.cc:2804] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.018388 25466 raft_consensus.cc:697] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.017879 25715 raft_consensus.cc:697] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.018596 25466 consensus_queue.cc:237] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.018837 25715 consensus_queue.cc:237] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:38.022833 23333 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 66777.8us
I20251023 20:15:38.023316 23333 leader_election.cc:304] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
W20251023 20:15:38.023567 23333 leader_election.cc:343] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1822fdf1f5f44dd4a742aa3c2f8443cf
I20251023 20:15:38.023721 23333 leader_election.cc:304] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:38.023892 23715 heartbeater.cc:499] Master 127.22.127.188:41417 was elected leader, sending a full tablet report...
I20251023 20:15:38.023955 23333 leader_election.cc:304] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters: ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.024351 24821 raft_consensus.cc:2804] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.024457 24821 raft_consensus.cc:697] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:38.024482 23333 leader_election.cc:304] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:38.025018 25167 raft_consensus.cc:493] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.025154 25167 raft_consensus.cc:515] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.025664 25167 leader_election.cc:290] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.024663 24821 consensus_queue.cc:237] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.026331 23333 leader_election.cc:304] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:38.026538 25489 raft_consensus.cc:2804] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.026634 25489 raft_consensus.cc:697] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:38.024837 25725 raft_consensus.cc:2804] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.026839 25489 consensus_queue.cc:237] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.026949 25725 raft_consensus.cc:697] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:38.027150 25725 consensus_queue.cc:237] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.027660 25726 raft_consensus.cc:2804] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.027771 25726 raft_consensus.cc:697] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:38.027961 25726 consensus_queue.cc:237] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.024905 25724 raft_consensus.cc:2804] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.028174 25724 raft_consensus.cc:697] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:38.028365 25724 consensus_queue.cc:237] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.033447 23787 tablet_bootstrap.cc:492] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.033593 23787 ts_tablet_manager.cc:1403] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20251023 20:15:38.034251 23787 raft_consensus.cc:359] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.034420 23787 raft_consensus.cc:385] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.034469 23787 raft_consensus.cc:740] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.034650 23787 consensus_queue.cc:260] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.035127 23787 ts_tablet_manager.cc:1434] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:38.035180 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 374) took 4506 ms. Trace:
I20251023 20:15:38.035388 23787 tablet_bootstrap.cc:492] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.036440 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 177) took 9255 ms. Trace:
I20251023 20:15:38.036588 24780 rpcz_store.cc:276] 1023 20:15:28.781006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.781048 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:37.877739 (+9096691us) ts_tablet_manager.cc:1151] Deleting tablet 79e3d422a66f432789c0cf4a00e57af2
1023 20:15:37.903898 (+ 26159us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.907747 (+ 3849us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:37.997501 (+ 89754us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.036419 (+ 38918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9096493}
I20251023 20:15:38.036901 23787 tablet_bootstrap.cc:654] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.037045 24780 tablet_replica.cc:333] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.037263 24297 raft_consensus.cc:493] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.037353 24780 raft_consensus.cc:2243] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.037494 24780 raft_consensus.cc:2272] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.037461 24297 raft_consensus.cc:515] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.038105 24780 ts_tablet_manager.cc:1916] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.038312 24297 leader_election.cc:290] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.039706 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet d62dbf044acd4459b26d2e7e50f6206e (table default.Table_8e0012945b9e401793b99da25bc5d76c [id=a0f3574408d0493e898036800ff93a2f]) successfully deleted
W20251023 20:15:38.041318 23203 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cc488f0beec14d0eb869b7474ec16205 (table default.Table_571807db7b4b4d189e39cd9c352f33c6 [id=134422fb6bdb4f9e850e477effc58d28], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cc488f0beec14d0eb869b7474ec16205 consensus state has no leader
I20251023 20:15:38.040567 23227 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae5a310b4a514c0f812bda133c756062" candidate_term: 3 candidate_status { last_received { term: 2 index: 807 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a" is_pre_election: true
I20251023 20:15:38.043115 23229 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 807 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a" is_pre_election: true
I20251023 20:15:38.044659 25492 ts_tablet_manager.cc:1929] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.044823 25492 log.cc:1199] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/05933cb9433d48d285b2db82b289bdc8
I20251023 20:15:38.045320 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet b3150a8478c144d285e999d5c9270fe0 (table default.Table_bfabdf6a28e645d697a566e89f2d79e3 [id=2db1100447a44e7fb43e87cdf7247d14]). Expected version 1 got 0
I20251023 20:15:38.045607 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet aa59141039de402e9ccce427604ecb31 (table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]). Expected version 1 got 0
I20251023 20:15:38.045967 23212 catalog_manager.cc:5741] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961) does not have the latest schema for tablet c7a63d37788240f0a0c3dbfdd14edeac (table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]). Expected version 2 got 0
I20251023 20:15:38.047276 23228 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6545140d94b7485aa81c51ad233355cd" candidate_term: 3 candidate_status { last_received { term: 2 index: 807 } } ignore_live_leader: false dest_uuid: "690ffde75cd745b6a87925967d54471a"
I20251023 20:15:38.047982 23204 catalog_manager.cc:4985] TS ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): tablet c9bfbf0acdc84f2f8fe553694ee9897c (table default.Table_a0edef92f99c4edebe22ae048c5dcdde [id=e0dec011bafc4166bdd335a6e94475d9]) successfully deleted
I20251023 20:15:38.048494 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.048687 23660 raft_consensus.cc:2468] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.048949 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 76380b341a304d6ba31648fbd1548f05 (table default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]) successfully deleted
I20251023 20:15:38.049144 23204 catalog_manager.cc:4985] TS 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): tablet 8317a54a5f7b4afe8da9d336268efe43 (table default.Table_1e7d5f5e8c224fa1b5786e0a40e3188a [id=273447c687354c109e50ecd2bd95084b]) successfully deleted
W20251023 20:15:38.049407 23205 catalog_manager.cc:4952] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): delete failed for tablet e5368813164949ff80e27bf59dcc931a because the tablet was not found. No further retry: Not found: Tablet not found: e5368813164949ff80e27bf59dcc931a
I20251023 20:15:38.049600 23470 leader_election.cc:304] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.049625 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59598ea40b9e4e5da0ec36f35f1094d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.049749 23666 raft_consensus.cc:3060] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.049855 23205 catalog_manager.cc:4985] TS d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): tablet 252b5480222f49439052c86a4115cc9b (table default.Table_bdb5b8891a814c998413f5b7a0a605cc [id=0674b989e3f74438b47e1fc653f5db49]) successfully deleted
I20251023 20:15:38.049948 25167 raft_consensus.cc:2804] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.050024 25167 raft_consensus.cc:493] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.050081 25167 raft_consensus.cc:3060] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.050228 23038 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu with pid 23185
I20251023 20:15:38.053226 25167 raft_consensus.cc:515] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.053503 25739 consensus_queue.cc:1048] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.054374 25167 leader_election.cc:290] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.055290 25492 ts_tablet_manager.cc:1950] T 05933cb9433d48d285b2db82b289bdc8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:38.055639 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 405) took 2753 ms. Trace:
I20251023 20:15:38.055756 25492 rpcz_store.cc:276] 1023 20:15:35.301838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.301873 (+ 35us) service_pool.cc:225] Handling call
1023 20:15:37.874446 (+2572573us) ts_tablet_manager.cc:1151] Deleting tablet 05933cb9433d48d285b2db82b289bdc8
1023 20:15:37.874455 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.957485 (+ 83030us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.044548 (+ 87063us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.055624 (+ 11076us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2572433}
I20251023 20:15:38.056268 25492 tablet_replica.cc:333] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:38.056496 25492 raft_consensus.cc:2243] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.056823 25492 raft_consensus.cc:2272] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.057394 25492 ts_tablet_manager.cc:1916] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.057590 23666 raft_consensus.cc:2468] T 59598ea40b9e4e5da0ec36f35f1094d6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.035346 25282 rpcz_store.cc:276] 1023 20:15:33.528240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.540009 (+ 11769us) service_pool.cc:225] Handling call
1023 20:15:37.924662 (+4384653us) ts_tablet_manager.cc:1151] Deleting tablet 3befdfc85bca42bb816e11d1dd832c8c
1023 20:15:37.924675 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:37.940437 (+ 15762us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.004457 (+ 64020us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.035159 (+ 30702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4384502}
I20251023 20:15:38.061803 25282 tablet_replica.cc:333] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:38.062036 25282 raft_consensus.cc:2243] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.062207 25282 raft_consensus.cc:2272] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.062865 25282 ts_tablet_manager.cc:1916] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.073173 25276 raft_consensus.cc:2749] T 3a74b3f22b854c1ab2b6cc817b84cf47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.073551 23604 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 340042us
I20251023 20:15:38.074103 25276 raft_consensus.cc:493] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.074213 24831 leader_election.cc:304] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae5a310b4a514c0f812bda133c756062; no voters: 6545140d94b7485aa81c51ad233355cd, 690ffde75cd745b6a87925967d54471a
I20251023 20:15:38.074221 25276 raft_consensus.cc:515] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.074343 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664]
I20251023 20:15:38.074786 25276 leader_election.cc:290] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.075235 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.075383 23663 raft_consensus.cc:3060] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:38.075757 23470 leader_election.cc:343] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 9a6a2d2b25824dbdafaf2a9cabb0f061
I20251023 20:15:38.075776 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0079496170849c6b3c233e1cd78e40e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.075984 23470 leader_election.cc:304] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.076072 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a45d9b3031b46d390e0a1010489c718" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.076114 25776 raft_consensus.cc:493] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.076210 25776 raft_consensus.cc:515] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.076404 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a216aeebe649f79b6d873e1c59df37" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.076654 25776 leader_election.cc:290] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:38.076830 23605 leader_election.cc:343] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.076633 25167 raft_consensus.cc:2804] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.077016 25776 raft_consensus.cc:493] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.077061 25167 raft_consensus.cc:697] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.077023 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95a06da281194f95b126483314f6debd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.077134 25776 raft_consensus.cc:515] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.077318 25167 consensus_queue.cc:237] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.077557 25776 leader_election.cc:290] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:38.077570 23605 leader_election.cc:343] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.077888 23663 raft_consensus.cc:2468] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.078083 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "118be7aa6380469db091d2788472e62b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.078358 24925 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 3 FOLLOWER]: Rejecting Update request from peer 690ffde75cd745b6a87925967d54471a for earlier term 2. Current term is 3. Ops: []
W20251023 20:15:38.078648 23605 leader_election.cc:343] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.080049 23494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 122) took 1344 ms. Trace:
I20251023 20:15:38.080127 23494 rpcz_store.cc:276] 1023 20:15:36.735751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.735815 (+ 64us) service_pool.cc:225] Handling call
1023 20:15:37.872729 (+1136914us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x6160002c0854. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.921364 (+ 48635us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x616000132754. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.990287 (+ 68923us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x616000198d54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.080006 (+ 89719us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2733397376}
I20251023 20:15:38.081885 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_191b94017cca4cfb946f38ab19ebfe20 [id=330c829b79fd4cce881fc565a3463863]
I20251023 20:15:38.082795 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_80e9f701dbaf4b009a1232cce62e2741 [id=3318730a93914fffb7029a880f262dc9]
I20251023 20:15:38.083190 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_b9e2bcb23af94936a9fbfd0817d9a51e [id=3528c0390b7240e8a565945035555dfa]
I20251023 20:15:38.083546 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_39fdd0d19cae4e928fdee07e74900b4b [id=356410224c0f44bdacbaa543e9dee27d]
I20251023 20:15:38.083922 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 [id=3973b56caa79477f8542118421741084]
I20251023 20:15:38.084280 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573]
I20251023 20:15:38.084838 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_8dc2f44cc1fc43e4a9851a0587a74edb [id=3bf5870046244dcd817441dce750a1e2]
W20251023 20:15:38.087031 23471 leader_election.cc:343] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.087550 23471 leader_election.cc:304] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.087857 25739 raft_consensus.cc:2749] T 60e8942deb7544a09b21c3d1c2a3f99f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.089380 25776 raft_consensus.cc:493] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.089514 25776 raft_consensus.cc:515] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.090034 25776 leader_election.cc:290] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.090266 23604 leader_election.cc:304] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.090507 25276 raft_consensus.cc:2804] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.090636 25276 raft_consensus.cc:697] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Running, Role: LEADER
I20251023 20:15:38.090863 25276 consensus_queue.cc:237] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.093719 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea0cbd6bec674cf5831486f500cc7b36" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.094280 23605 leader_election.cc:343] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.099115 24297 consensus_queue.cc:1048] T 5161336598884742ad403fc88c9d9f0b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.103770 23604 leader_election.cc:343] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.103955 23604 leader_election.cc:304] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.104218 25776 raft_consensus.cc:2749] T a0079496170849c6b3c233e1cd78e40e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.104590 23604 leader_election.cc:343] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.104748 23604 leader_election.cc:304] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.104977 25776 raft_consensus.cc:2749] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.108227 25776 raft_consensus.cc:493] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.108340 25776 raft_consensus.cc:515] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.088145 23471 leader_election.cc:304] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:38.094182 25739 raft_consensus.cc:493] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.108600 25739 raft_consensus.cc:515] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.108856 25776 leader_election.cc:290] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.109067 23471 leader_election.cc:304] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.109225 25739 leader_election.cc:290] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.109387 25739 raft_consensus.cc:2804] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.109531 25739 raft_consensus.cc:697] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.109505 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63ef1bae87c54aa988dfe49f828fe4f8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.109831 25739 consensus_queue.cc:237] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:38.110329 23605 leader_election.cc:343] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.110857 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3befdfc85bca42bb816e11d1dd832c8c" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.100453 23470 leader_election.cc:304] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.111850 25740 raft_consensus.cc:2749] T cf38936bb12846bd81a2c79ffbd18ccb P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.112133 25740 raft_consensus.cc:2804] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.112349 25740 raft_consensus.cc:697] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.112314 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbab01ecb27a426188689408079eaa01" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.112473 23660 raft_consensus.cc:2468] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.112663 25740 consensus_queue.cc:237] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.113258 23470 leader_election.cc:304] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:38.113045 24831 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.188:41417, user_credentials={real_user=slave}} blocked reactor thread for 61737.4us
I20251023 20:15:38.113778 25466 raft_consensus.cc:2804] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.113893 25466 raft_consensus.cc:493] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.113996 25466 raft_consensus.cc:3060] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.115835 25822 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251023 20:15:38.099210 23333 leader_election.cc:304] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:38.117419 25726 raft_consensus.cc:2804] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.117539 25726 raft_consensus.cc:697] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
I20251023 20:15:38.117615 23333 leader_election.cc:304] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.117880 25726 consensus_queue.cc:237] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.118124 25592 raft_consensus.cc:2749] T 2202632a57514c849760509360113078 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.119045 23333 leader_election.cc:304] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.119393 24297 raft_consensus.cc:2749] T cd548fd9de284377ab557bd98f621068 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.120080 23333 consensus_peers.cc:597] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fa6a838b1bd409ba2a4e2074576ee23. This is attempt 6: this message will repeat every 5th retry.
I20251023 20:15:38.121515 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bce63866d524067b87145b07347ffcf" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.121834 23389 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "857da3642f0846ce92aaed0491e89644" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.121963 23389 raft_consensus.cc:3060] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.122077 23392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a6a2d2b25824dbdafaf2a9cabb0f061" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.122239 23392 raft_consensus.cc:3060] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.122395 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59598ea40b9e4e5da0ec36f35f1094d6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.124573 23389 raft_consensus.cc:2468] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.124807 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d227b45bfcc64d76a51cff347e6e1486" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.124926 23399 raft_consensus.cc:3060] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.125401 23392 raft_consensus.cc:2468] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.127269 23399 raft_consensus.cc:2468] T d227b45bfcc64d76a51cff347e6e1486 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.127635 23391 raft_consensus.cc:3060] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.130088 23391 raft_consensus.cc:2468] T 4bce63866d524067b87145b07347ffcf P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.130352 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d145a33b964a4ca1f0001ac6cf990b" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.146765 25740 consensus_queue.cc:1048] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.152278 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.152577 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.153733 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbab01ecb27a426188689408079eaa01" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.153911 25276 raft_consensus.cc:493] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.154152 25276 raft_consensus.cc:515] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:38.155932 23471 leader_election.cc:343] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.156786 23471 leader_election.cc:304] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:38.157630 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ac4b704784f2423a82ad591bbed856ab [id=3cf9ba4080ce4e5ead110ed71e43ce1f]
I20251023 20:15:38.157912 23471 leader_election.cc:304] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32
I20251023 20:15:38.158208 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_b029eee1964748f3bb848b0f6309a357 [id=3d3676ef16f540c190331ed14e25d985]
I20251023 20:15:38.158639 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_698f8853c4774ccdae2e37d28f90f368 [id=3e597b14c5d944838c32914dc1779d20]
I20251023 20:15:38.158658 23471 leader_election.cc:304] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
W20251023 20:15:38.159338 23471 leader_election.cc:343] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.159463 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7824bcf5ff034be78ed7dee87e4cd78d [id=44239d72fdcd48258921b9490c822137]
W20251023 20:15:38.159734 23471 leader_election.cc:343] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.159880 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e177ffb3adcc48dc84739ca2bdc1ae6e [id=44d2a2e9d5dc49be9b9474da1b1841bc]
W20251023 20:15:38.160202 23471 leader_election.cc:343] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.160269 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_bdcbe2f8128a4a89b305439e5123c2fe [id=452652d5f027441cb6f821e5d1db0c03]
I20251023 20:15:38.160626 25741 consensus_queue.cc:1048] T dae906f257c346d7a1b28c9052d600b5 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.160900 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_58a322a6e70c4cd58aa1cca39508e78d [id=47b905f0341d43c588d84ebd16eaa21d]
I20251023 20:15:38.161399 25741 raft_consensus.cc:2804] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.161526 25741 raft_consensus.cc:697] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
W20251023 20:15:38.161515 23333 leader_election.cc:343] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 3befdfc85bca42bb816e11d1dd832c8c
I20251023 20:15:38.161623 23333 leader_election.cc:304] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.161782 25741 consensus_queue.cc:237] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.162622 25595 raft_consensus.cc:2749] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.162810 25470 raft_consensus.cc:2804] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.162959 25470 raft_consensus.cc:697] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.163177 25470 consensus_queue.cc:237] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.163514 25492 ts_tablet_manager.cc:1929] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.163646 25492 log.cc:1199] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a86a25659ec44d7c825783036342ed45
I20251023 20:15:38.164036 25492 ts_tablet_manager.cc:1950] T a86a25659ec44d7c825783036342ed45 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:38.164213 25596 raft_consensus.cc:2749] T c3d145a33b964a4ca1f0001ac6cf990b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.164361 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 407) took 2862 ms. Trace:
I20251023 20:15:38.164462 25492 rpcz_store.cc:276] 1023 20:15:35.302042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.302077 (+ 35us) service_pool.cc:225] Handling call
1023 20:15:38.056223 (+2754146us) ts_tablet_manager.cc:1151] Deleting tablet a86a25659ec44d7c825783036342ed45
1023 20:15:38.056233 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.115966 (+ 59733us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.144753 (+ 28787us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.164315 (+ 19562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2754023}
I20251023 20:15:38.164836 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 408) took 2862 ms. Trace:
I20251023 20:15:38.164945 25492 rpcz_store.cc:276] 1023 20:15:35.302130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.302224 (+ 94us) service_pool.cc:225] Handling call
1023 20:15:38.164723 (+2862499us) ts_tablet_manager.cc:1151] Deleting tablet a8eaa2167f82475ca2f6a110aba288ec
1023 20:15:38.164734 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.164795 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2862331}
I20251023 20:15:38.165236 25492 tablet_replica.cc:333] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:38.165458 25492 raft_consensus.cc:2243] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.165820 25492 raft_consensus.cc:2272] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.170113 24925 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 3 FOLLOWER]: Refusing update from remote peer 6545140d94b7485aa81c51ad233355cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 807. Preceding OpId from leader: term: 3 index: 808. (index mismatch)
I20251023 20:15:38.170840 25770 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 808, Last known committed idx: 807, Time since last communication: 0.000s
I20251023 20:15:38.171204 23786 tablet_bootstrap.cc:492] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.171373 23786 ts_tablet_manager.cc:1403] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.002s
I20251023 20:15:38.172170 23786 raft_consensus.cc:359] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.172362 23786 raft_consensus.cc:385] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.172441 23786 raft_consensus.cc:740] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.172708 23786 consensus_queue.cc:260] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.173241 23786 ts_tablet_manager.cc:1434] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:38.173563 23786 tablet_bootstrap.cc:492] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:38.174549 23604 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.130:34173, user_credentials={real_user=slave}} blocked reactor thread for 69958.2us
I20251023 20:15:38.175314 25776 raft_consensus.cc:493] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.175441 25776 raft_consensus.cc:515] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.175959 25776 leader_election.cc:290] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.176084 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a216aeebe649f79b6d873e1c59df37" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.177204 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95a06da281194f95b126483314f6debd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.177541 25466 raft_consensus.cc:515] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.177809 23790 tablet_bootstrap.cc:492] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:38.178488 23605 leader_election.cc:343] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 8d9b192c111c47d1983e9a48d07da288
I20251023 20:15:38.178903 23790 ts_tablet_manager.cc:1403] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.228s user 0.003s sys 0.008s
I20251023 20:15:38.179805 23790 raft_consensus.cc:359] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.181149 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "118be7aa6380469db091d2788472e62b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.181563 23790 raft_consensus.cc:385] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.181677 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea0cbd6bec674cf5831486f500cc7b36" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.181915 25466 leader_election.cc:290] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.181978 23790 raft_consensus.cc:740] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.182122 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63ef1bae87c54aa988dfe49f828fe4f8" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.182576 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f2a4f1d52984af58d6286671389b6a9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.183010 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9b192c111c47d1983e9a48d07da288" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.182729 23790 consensus_queue.cc:260] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.183944 23790 ts_tablet_manager.cc:1434] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251023 20:15:38.184911 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbab01ecb27a426188689408079eaa01" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.185106 23660 raft_consensus.cc:3060] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.188021 23660 raft_consensus.cc:2468] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.193372 23790 tablet_bootstrap.cc:492] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.193548 23470 leader_election.cc:304] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.194031 25596 raft_consensus.cc:2804] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.194165 25596 raft_consensus.cc:697] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.194456 25596 consensus_queue.cc:237] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.194921 23790 tablet_bootstrap.cc:654] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.200583 25776 raft_consensus.cc:493] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.200727 25776 raft_consensus.cc:515] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.201388 25776 leader_election.cc:290] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.201495 25822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6545140d94b7485aa81c51ad233355cd. Latest consensus state: current_term: 3 leader_uuid: "6545140d94b7485aa81c51ad233355cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:38.201741 25822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: FOLLOWER
W20251023 20:15:38.204209 23604 leader_election.cc:343] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.204396 23604 leader_election.cc:304] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.204654 25776 raft_consensus.cc:2749] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.205428 23604 leader_election.cc:343] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:38.205474 23605 leader_election.cc:343] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 1e19859a65a144598e1612be89daa3a1
I20251023 20:15:38.206319 25566 raft_consensus.cc:493] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.206507 25566 raft_consensus.cc:515] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.207163 25566 leader_election.cc:290] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.208813 25276 leader_election.cc:290] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.211092 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e19859a65a144598e1612be89daa3a1" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.211858 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757d41e027ba461ea051b38a2d685407" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.212046 23660 raft_consensus.cc:2468] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.213101 23604 leader_election.cc:304] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.213333 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f2a4f1d52984af58d6286671389b6a9" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.213575 25776 raft_consensus.cc:2749] T 95a06da281194f95b126483314f6debd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.213853 23605 leader_election.cc:343] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.214483 23470 leader_election.cc:304] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.217330 25739 raft_consensus.cc:2804] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.217428 25739 raft_consensus.cc:493] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.217528 25739 raft_consensus.cc:3060] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:38.219345 23604 leader_election.cc:343] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.219450 23604 leader_election.cc:304] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.219764 25776 raft_consensus.cc:2749] T 118be7aa6380469db091d2788472e62b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.220124 23604 leader_election.cc:343] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.220211 23604 leader_election.cc:304] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.220388 25776 raft_consensus.cc:2749] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.220486 23787 tablet_bootstrap.cc:492] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
W20251023 20:15:38.220677 23604 leader_election.cc:343] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.220690 23787 ts_tablet_manager.cc:1403] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.185s user 0.010s sys 0.022s
I20251023 20:15:38.220772 23604 leader_election.cc:304] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.221781 25776 raft_consensus.cc:2749] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.222139 23604 leader_election.cc:343] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.222221 23787 raft_consensus.cc:359] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.222582 23787 raft_consensus.cc:385] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.222692 23787 raft_consensus.cc:740] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.223045 23787 consensus_queue.cc:260] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.224045 23787 ts_tablet_manager.cc:1434] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:38.225135 23787 tablet_bootstrap.cc:492] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.233801 23787 tablet_bootstrap.cc:654] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:38.235036 23333 consensus_peers.cc:597] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05933cb9433d48d285b2db82b289bdc8. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:38.236022 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbab01ecb27a426188689408079eaa01" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.222220 23604 leader_election.cc:304] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.236994 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757d41e027ba461ea051b38a2d685407" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.237112 25776 raft_consensus.cc:2749] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.238615 23471 leader_election.cc:343] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.239293 23786 tablet_bootstrap.cc:654] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.239813 25167 consensus_queue.cc:1048] T 1e343a7d39cf4329b99c9f9ffd8cefb2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.239493 23604 leader_election.cc:343] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:38.240513 23471 leader_election.cc:343] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.240434 23604 leader_election.cc:304] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.241264 25739 raft_consensus.cc:515] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.241914 25776 raft_consensus.cc:2749] T 8d9b192c111c47d1983e9a48d07da288 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.244832 23492 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 120) took 2031 ms. Trace:
I20251023 20:15:38.251421 24821 consensus_queue.cc:1048] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.252147 23492 rpcz_store.cc:276] 1023 20:15:36.213254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.213313 (+ 59us) service_pool.cc:225] Handling call
1023 20:15:37.347335 (+1134022us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x616000161b54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.779853 (+432518us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x6160002d2254. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.899406 (+119553us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x616000155254. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.921717 (+ 22311us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x6160000c4354. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.992254 (+ 70537us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x616000173e54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.168273 (+176019us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x616000114a54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.219864 (+ 51591us) spinlock_profiling.cc:243] Waited 50.5 ms on lock 0x616000198d54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.242175 (+ 22311us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x6160001c9954. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.244786 (+ 2611us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4552804608}
I20251023 20:15:38.252030 23493 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 119) took 2195 ms. Trace:
I20251023 20:15:38.253269 25822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6545140d94b7485aa81c51ad233355cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:38.248977 23496 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 123) took 1482 ms. Trace:
I20251023 20:15:38.253476 25822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [sys.catalog]: This master's current role is: FOLLOWER
I20251023 20:15:38.253592 23496 rpcz_store.cc:276] 1023 20:15:36.766830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.766893 (+ 63us) service_pool.cc:225] Handling call
1023 20:15:37.872125 (+1105232us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x6160002c0854. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.920733 (+ 48608us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x616000132754. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.991082 (+ 70349us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x616000173e54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.166798 (+175716us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x616000114a54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.222573 (+ 55775us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x616000198d54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.245070 (+ 22497us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x6160001c9954. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.248928 (+ 3858us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3266149376}
I20251023 20:15:38.254698 23493 rpcz_store.cc:276] 1023 20:15:36.052644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.052715 (+ 71us) service_pool.cc:225] Handling call
1023 20:15:37.901224 (+1848509us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x616000155254. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.971490 (+ 70266us) spinlock_profiling.cc:243] Waited 67 ms on lock 0x6160000d6354. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.167489 (+195999us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x616000114a54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.222863 (+ 55374us) spinlock_profiling.cc:243] Waited 49.8 ms on lock 0x616000198d54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.243928 (+ 21065us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x6160001c9954. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.248041 (+ 4113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4998196608}
I20251023 20:15:38.251297 23495 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:44698 (request call id 121) took 1623 ms. Trace:
I20251023 20:15:38.256835 23495 rpcz_store.cc:276] 1023 20:15:36.627693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:36.627743 (+ 50us) service_pool.cc:225] Handling call
1023 20:15:37.916027 (+1288284us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x6160001ab654. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:37.990986 (+ 74959us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x616000173e54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.166195 (+175209us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x616000114a54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.222585 (+ 56390us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x616000198d54. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.241655 (+ 19070us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x6160001c9954. stack: 00007fc3982583e6 00007fc3982581a0 00007fc39723629d 0000000000983974 000000000098393d 000000000098330a 00007fc3a40d1020 00007fc3a7f811ef 00007fc3a239ca56 00007fc3a239c785 00007fc39f1cdf1e 00007fc39f1ccdfe 00007fc39f1d3035 00007fc39f1d395f 00007fc39f1d3788 0000000000a880ec
1023 20:15:38.251262 (+ 9607us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3642165248}
W20251023 20:15:38.257705 23604 leader_election.cc:343] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.258466 23604 leader_election.cc:304] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.259964 25776 raft_consensus.cc:2749] T 1e19859a65a144598e1612be89daa3a1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.260378 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757d41e027ba461ea051b38a2d685407" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.260543 23660 raft_consensus.cc:3060] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.262351 25749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6545140d94b7485aa81c51ad233355cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } } }
I20251023 20:15:38.262606 25749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd [sys.catalog]: This master's current role is: LEADER
I20251023 20:15:38.263245 23660 raft_consensus.cc:2468] T 757d41e027ba461ea051b38a2d685407 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.264220 23470 leader_election.cc:304] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.266978 25167 raft_consensus.cc:2804] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.267133 25167 raft_consensus.cc:697] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.267405 25167 consensus_queue.cc:237] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.272372 25739 leader_election.cc:290] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:38.280300 23470 consensus_peers.cc:597] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3befdfc85bca42bb816e11d1dd832c8c. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:38.286402 23787 tablet_bootstrap.cc:492] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.286674 23787 ts_tablet_manager.cc:1403] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.002s
I20251023 20:15:38.287896 23787 raft_consensus.cc:359] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.288461 23787 raft_consensus.cc:385] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.288643 23787 raft_consensus.cc:740] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.289053 23787 consensus_queue.cc:260] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.290092 23787 ts_tablet_manager.cc:1434] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:38.290503 25282 ts_tablet_manager.cc:1929] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:38.290580 23787 tablet_bootstrap.cc:492] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.291356 24780 ts_tablet_manager.cc:1929] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.291584 24780 log.cc:1199] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/4b6c3e7169154e3497dd3570dca0d745
I20251023 20:15:38.292011 24780 ts_tablet_manager.cc:1950] T 4b6c3e7169154e3497dd3570dca0d745 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.292831 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 178) took 9480 ms. Trace:
I20251023 20:15:38.293009 24780 rpcz_store.cc:276] 1023 20:15:28.812140 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.812734 (+ 594us) service_pool.cc:225] Handling call
1023 20:15:38.036945 (+9224211us) ts_tablet_manager.cc:1151] Deleting tablet 4b6c3e7169154e3497dd3570dca0d745
1023 20:15:38.037005 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.111846 (+ 74841us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.291241 (+179395us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.292792 (+ 1551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9223934}
I20251023 20:15:38.290669 25282 log.cc:1199] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/db758a7a7a89410f8ac1da72048fbc56
I20251023 20:15:38.293445 24780 tablet_replica.cc:333] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.293696 24780 raft_consensus.cc:2243] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.293948 24780 raft_consensus.cc:2272] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.294512 24780 ts_tablet_manager.cc:1916] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.295980 25282 ts_tablet_manager.cc:1950] T db758a7a7a89410f8ac1da72048fbc56 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:38.304482 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 377) took 4774 ms. Trace:
I20251023 20:15:38.307009 23787 tablet_bootstrap.cc:654] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.305452 25282 rpcz_store.cc:276] 1023 20:15:33.529906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.540186 (+ 10280us) service_pool.cc:225] Handling call
1023 20:15:38.061753 (+4521567us) ts_tablet_manager.cc:1151] Deleting tablet db758a7a7a89410f8ac1da72048fbc56
1023 20:15:38.061765 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.134932 (+ 73167us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.290392 (+155460us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.304174 (+ 13782us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4521447}
I20251023 20:15:38.309378 25492 ts_tablet_manager.cc:1916] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.309698 25282 tablet_replica.cc:333] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:38.313452 25282 raft_consensus.cc:2243] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.315346 25282 raft_consensus.cc:2272] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.317443 25739 raft_consensus.cc:493] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.317613 25739 raft_consensus.cc:515] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.318161 25739 leader_election.cc:290] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.318670 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.318832 23660 raft_consensus.cc:2468] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.319424 23470 leader_election.cc:304] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.319692 25167 raft_consensus.cc:2804] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.319784 25167 raft_consensus.cc:493] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.319868 25167 raft_consensus.cc:3060] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.320003 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_95fb2c2528f44d8d84f4cd67d9663502 [id=4867e660fb23438495400e364dba7b48]
I20251023 20:15:38.320549 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3448a0da3d5b4fffa69b2f473d4416b2 [id=492b5c03c69c46e3afcb57168577a59b]
I20251023 20:15:38.320983 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_5e2be9d05d7d431685748f79ce1c7311 [id=494f3980030c4d1fa22fec4ddcf14ff8]
I20251023 20:15:38.321758 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_331b508de7624556842102252e4ccc38 [id=4a9c4e7500ab40f5b0daf197ce160eb5]
I20251023 20:15:38.322202 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_240b16ca6e1543f2aab607c1d6a447fd [id=4aa2d573bbb34a3ebda2ac6376a387ea]
I20251023 20:15:38.322621 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ababfc027cb24714ab70240be6fae889 [id=4b8e127e7d37433e8cfe495eda2bc815]
I20251023 20:15:38.325253 25282 ts_tablet_manager.cc:1916] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.330097 25167 raft_consensus.cc:515] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.331080 25167 leader_election.cc:290] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.300611 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "757d41e027ba461ea051b38a2d685407" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.317826 24297 consensus_queue.cc:1048] T 8d22aa40e5844d78a8679a3ad1b08da9 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.332459 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.332594 24297 raft_consensus.cc:493] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.332660 23667 raft_consensus.cc:3060] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.332747 24297 raft_consensus.cc:515] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } }
I20251023 20:15:38.333356 24297 leader_election.cc:290] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211), ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173)
I20251023 20:15:38.333595 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.334506 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.335292 23667 raft_consensus.cc:2468] T a7f9428277fb4ebf989fed7569dee687 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.337055 23470 leader_election.cc:304] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.338389 23786 tablet_bootstrap.cc:492] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.338587 23786 ts_tablet_manager.cc:1403] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.165s user 0.007s sys 0.013s
I20251023 20:15:38.339356 23786 raft_consensus.cc:359] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.339568 23786 raft_consensus.cc:385] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.339694 23786 raft_consensus.cc:740] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.340011 23786 consensus_queue.cc:260] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.340555 23786 ts_tablet_manager.cc:1434] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:38.340736 25167 raft_consensus.cc:2804] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.341107 25167 raft_consensus.cc:697] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.340617 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_05a9ffe8a73a4527be8ac2ba2fecc052 [id=4bbc0f95fee64c7c8570961ba3f1c0a6]
I20251023 20:15:38.341470 25167 consensus_queue.cc:237] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.341796 23787 tablet_bootstrap.cc:492] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.341993 23787 ts_tablet_manager.cc:1403] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20251023 20:15:38.342330 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_cd166f9dbcdb47ad988e2d387a6a1c5e [id=4c679b7dd63d4db2bb3ea41860cf80b5]
I20251023 20:15:38.344633 23787 raft_consensus.cc:359] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.344899 23787 raft_consensus.cc:385] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.345019 23787 raft_consensus.cc:740] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.345364 23787 consensus_queue.cc:260] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.348157 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_6443f81020254f45a1b7fd93ad217e09 [id=4d43a0eda13041689a414ecbadb85e09]
I20251023 20:15:38.350004 25492 ts_tablet_manager.cc:1929] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.350579 23787 ts_tablet_manager.cc:1434] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251023 20:15:38.351092 23787 tablet_bootstrap.cc:492] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.356343 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d55345cd0dd44edaa55e9e412037691f [id=4da4a7df78da46b18a39004985932e04]
I20251023 20:15:38.359581 25776 raft_consensus.cc:493] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:38.362720 23333 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:38.367652 25776 raft_consensus.cc:515] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.368352 25776 leader_election.cc:290] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:38.370209 23605 leader_election.cc:343] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 5acdc66df9c748a1a6300962f62fcf36
I20251023 20:15:38.371181 23787 tablet_bootstrap.cc:654] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.380452 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_baf8abd9a132412a8b53ac8d9e044dcf [id=51ea9f032fdd40bb9f0ea42bb80c5cbf]
I20251023 20:15:38.381017 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_b3b6a3c4f36f4f888a6b5c483002e6ba [id=52a26bb4b9384c829830c1fe988d04be]
I20251023 20:15:38.382005 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249]
I20251023 20:15:38.382766 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7417d01d91d44665a6f021a647db582c [id=580458858c744a5f98da3fa134aea223]
I20251023 20:15:38.382926 25282 ts_tablet_manager.cc:1929] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:38.383679 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b [id=58cb342d7d6142858153064d66e3712c]
I20251023 20:15:38.386493 24780 ts_tablet_manager.cc:1929] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.386703 24780 log.cc:1199] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/86b9f4c5d9e0469c87071a33722ce3eb
I20251023 20:15:38.386914 23790 tablet_bootstrap.cc:492] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:38.387109 23790 ts_tablet_manager.cc:1403] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.194s user 0.001s sys 0.005s
I20251023 20:15:38.350607 23786 tablet_bootstrap.cc:492] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.387324 25492 log.cc:1199] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/9ed0aa7e36e64843ad6b2ad4b415832a
I20251023 20:15:38.387692 25492 ts_tablet_manager.cc:1950] T 9ed0aa7e36e64843ad6b2ad4b415832a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:38.388479 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 402) took 3086 ms. Trace:
I20251023 20:15:38.388732 23786 tablet_bootstrap.cc:654] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.388689 25492 rpcz_store.cc:276] 1023 20:15:35.301507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.302322 (+ 815us) service_pool.cc:225] Handling call
1023 20:15:38.165170 (+2862848us) ts_tablet_manager.cc:1151] Deleting tablet 9ed0aa7e36e64843ad6b2ad4b415832a
1023 20:15:38.165179 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.312443 (+147264us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.349871 (+ 37428us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.388417 (+ 38546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2862724}
I20251023 20:15:38.389218 25492 tablet_replica.cc:333] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:38.389412 25492 raft_consensus.cc:2243] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:38.358266 23471 leader_election.cc:343] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.389650 25492 raft_consensus.cc:2272] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a [term 2 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:38.390285 23471 leader_election.cc:343] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:38.390713 23471 leader_election.cc:343] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.369037 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5acdc66df9c748a1a6300962f62fcf36" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:38.391441 23604 leader_election.cc:343] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.391570 23604 leader_election.cc:304] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.391781 23790 raft_consensus.cc:359] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.391983 25776 raft_consensus.cc:2749] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.392057 23790 raft_consensus.cc:385] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.392139 23790 raft_consensus.cc:740] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.392357 23790 consensus_queue.cc:260] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.393039 23790 ts_tablet_manager.cc:1434] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251023 20:15:38.393384 23790 tablet_bootstrap.cc:492] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.394868 23790 tablet_bootstrap.cc:654] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.399331 23787 tablet_bootstrap.cc:492] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.399573 23787 ts_tablet_manager.cc:1403] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.000s
W20251023 20:15:38.400617 23333 leader_election.cc:343] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Not found: Tablet not found: 1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:38.400767 23333 leader_election.cc:304] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.400995 23787 raft_consensus.cc:359] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.401352 23787 raft_consensus.cc:385] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.401369 24297 raft_consensus.cc:2749] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:38.401508 23787 raft_consensus.cc:740] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.401511 24821 consensus_queue.cc:1048] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.402244 23787 consensus_queue.cc:260] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.403311 23787 ts_tablet_manager.cc:1434] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:38.403903 23787 tablet_bootstrap.cc:492] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.409406 25282 log.cc:1199] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/fbab01ecb27a426188689408079eaa01
I20251023 20:15:38.410219 25282 ts_tablet_manager.cc:1950] T fbab01ecb27a426188689408079eaa01 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:38.415318 23786 tablet_bootstrap.cc:492] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.415501 23786 ts_tablet_manager.cc:1403] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.010s
I20251023 20:15:38.416951 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 379) took 4886 ms. Trace:
I20251023 20:15:38.417196 25282 rpcz_store.cc:276] 1023 20:15:33.530721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.540330 (+ 9609us) service_pool.cc:225] Handling call
1023 20:15:38.308951 (+4768621us) ts_tablet_manager.cc:1151] Deleting tablet fbab01ecb27a426188689408079eaa01
1023 20:15:38.309241 (+ 290us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.361850 (+ 52609us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.382811 (+ 20961us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.416905 (+ 34094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4768471}
I20251023 20:15:38.387161 24780 ts_tablet_manager.cc:1950] T 86b9f4c5d9e0469c87071a33722ce3eb P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.418665 23786 raft_consensus.cc:359] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.418979 23786 raft_consensus.cc:385] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.419066 23786 raft_consensus.cc:740] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.419281 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 381) took 4864 ms. Trace:
I20251023 20:15:38.419958 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 179) took 9606 ms. Trace:
I20251023 20:15:38.419329 23786 consensus_queue.cc:260] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.420087 24780 rpcz_store.cc:276] 1023 20:15:28.813245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.813506 (+ 261us) service_pool.cc:225] Handling call
1023 20:15:38.293387 (+9479881us) ts_tablet_manager.cc:1151] Deleting tablet 86b9f4c5d9e0469c87071a33722ce3eb
1023 20:15:38.293395 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.343615 (+ 50220us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.386357 (+ 42742us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.419905 (+ 33548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9479713}
I20251023 20:15:38.420387 24780 tablet_replica.cc:333] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.420611 24780 raft_consensus.cc:2243] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.420746 23786 ts_tablet_manager.cc:1434] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:38.420765 24780 raft_consensus.cc:2272] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.422197 24780 ts_tablet_manager.cc:1916] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.422230 23787 tablet_bootstrap.cc:654] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.425343 23786 tablet_bootstrap.cc:492] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.426568 25167 raft_consensus.cc:493] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.426784 25167 raft_consensus.cc:515] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.428153 25492 ts_tablet_manager.cc:1916] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.428049 25167 leader_election.cc:290] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.430011 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.430027 23333 consensus_peers.cc:597] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26a680ad3aed487593bd1b63004dbbfc. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:38.430437 23471 leader_election.cc:343] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.434000 23786 tablet_bootstrap.cc:654] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.435139 24780 ts_tablet_manager.cc:1929] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:38.435276 24780 log.cc:1199] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/1ae910ac41fd4e349f159ab27e44375a
I20251023 20:15:38.435611 24780 ts_tablet_manager.cc:1950] T 1ae910ac41fd4e349f159ab27e44375a P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.436694 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 180) took 9622 ms. Trace:
I20251023 20:15:38.436817 24780 rpcz_store.cc:276] 1023 20:15:28.813786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.813930 (+ 144us) service_pool.cc:225] Handling call
1023 20:15:38.420325 (+9606395us) ts_tablet_manager.cc:1151] Deleting tablet 1ae910ac41fd4e349f159ab27e44375a
1023 20:15:38.420335 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.431750 (+ 11415us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.435039 (+ 3289us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.436660 (+ 1621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9606235}
I20251023 20:15:38.437337 24780 tablet_replica.cc:333] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.440171 24780 raft_consensus.cc:2243] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.440887 24780 raft_consensus.cc:2272] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.441504 24780 ts_tablet_manager.cc:1916] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.420583 25282 rpcz_store.cc:276] 1023 20:15:33.555303 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.555378 (+ 75us) service_pool.cc:225] Handling call
1023 20:15:38.418607 (+4863229us) ts_tablet_manager.cc:1151] Deleting tablet b90810781f724c78935a33d868ffa27d
1023 20:15:38.418619 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.419261 (+ 642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4862774}
I20251023 20:15:38.448992 24780 ts_tablet_manager.cc:1929] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.428066 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.449185 24780 log.cc:1199] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/2fa6a838b1bd409ba2a4e2074576ee23
I20251023 20:15:38.449191 23667 raft_consensus.cc:2468] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.449585 24780 ts_tablet_manager.cc:1950] T 2fa6a838b1bd409ba2a4e2074576ee23 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.449599 23470 leader_election.cc:304] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.450055 25167 raft_consensus.cc:2804] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.450151 25167 raft_consensus.cc:493] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.450255 25167 raft_consensus.cc:3060] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.450474 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 181) took 9636 ms. Trace:
I20251023 20:15:38.450563 24780 rpcz_store.cc:276] 1023 20:15:28.814221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.814383 (+ 162us) service_pool.cc:225] Handling call
1023 20:15:38.437266 (+9622883us) ts_tablet_manager.cc:1151] Deleting tablet 2fa6a838b1bd409ba2a4e2074576ee23
1023 20:15:38.437276 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.445880 (+ 8604us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.448875 (+ 2995us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.450437 (+ 1562us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2075,"tablet-delete.queue_time_us":9622741}
I20251023 20:15:38.450995 24780 tablet_replica.cc:333] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.451179 24780 raft_consensus.cc:2243] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.451731 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_de4427f05119413b8983f4e818072f98 [id=591b99acc8dd4e7388acb07fdf88a50a]
I20251023 20:15:38.452195 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_6ea21ba3f1684634907e258298159baf [id=5962bfcedadc486d8f5e985d1c213fe2]
I20251023 20:15:38.452620 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_8587355637694585afe381e3e77c3958 [id=59862d7d13674aad83a3df0d07dce998]
I20251023 20:15:38.452736 25167 raft_consensus.cc:515] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.452841 24780 raft_consensus.cc:2272] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.453617 25167 leader_election.cc:290] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.454660 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.455219 25592 consensus_queue.cc:1048] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.455883 23471 leader_election.cc:343] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.464346 24780 ts_tablet_manager.cc:1916] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.464552 25776 raft_consensus.cc:493] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.464700 25776 raft_consensus.cc:515] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.465384 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d [id=5a207d264756433f94af5164462692ff]
I20251023 20:15:38.465736 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9749c6f3fe5545ab93bf49aae97e9818" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:38.466233 23604 leader_election.cc:343] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.466312 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_aeba123341a14c5aba5a1ccca37e5395 [id=5b8f061cbf014eed80a09b13ee85429f]
I20251023 20:15:38.466786 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37]
I20251023 20:15:38.467574 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_68a2dbd9e4cb46cbab9b4b9e49e64d15 [id=5f0b47b3930349658b7c8dbc7d9978a8]
I20251023 20:15:38.468221 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_0c73476a1a6d41eeb3ed876c729c6d21 [id=619c9b0e1ec84349a682a425cbe85087]
I20251023 20:15:38.468887 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad]
I20251023 20:15:38.469633 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485]
I20251023 20:15:38.473181 25492 ts_tablet_manager.cc:1929] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251023 20:15:38.474812 25276 raft_consensus.cc:493] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.474941 25276 raft_consensus.cc:515] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.475489 25276 leader_election.cc:290] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.476639 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44934f52cbeb448ba081b57cde783b2a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.477133 23605 leader_election.cc:343] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.478344 24780 ts_tablet_manager.cc:1929] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:38.482386 25276 raft_consensus.cc:493] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.482507 25276 raft_consensus.cc:515] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.485453 25489 raft_consensus.cc:493] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.485725 25489 raft_consensus.cc:515] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.490276 25489 leader_election.cc:290] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.490814 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.491014 23668 raft_consensus.cc:2393] T 08b8c7c1b50d495f9ed54b8bfa1e984d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:38.491190 25489 consensus_queue.cc:1048] T e9e2fec8f4e14036af714427481f2f36 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.492694 24780 log.cc:1199] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/9a4f012469fa4e8e827e485287ad76dd
I20251023 20:15:38.498065 24780 ts_tablet_manager.cc:1950] T 9a4f012469fa4e8e827e485287ad76dd P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.498943 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 182) took 9684 ms. Trace:
I20251023 20:15:38.499087 24780 rpcz_store.cc:276] 1023 20:15:28.814627 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.814761 (+ 134us) service_pool.cc:225] Handling call
1023 20:15:38.450938 (+9636177us) ts_tablet_manager.cc:1151] Deleting tablet 9a4f012469fa4e8e827e485287ad76dd
1023 20:15:38.450947 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.469061 (+ 18114us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.478223 (+ 9162us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.498903 (+ 20680us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1491,"tablet-delete.queue_time_us":9636013}
I20251023 20:15:38.499621 24780 tablet_replica.cc:333] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.499877 24780 raft_consensus.cc:2243] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.500276 24780 raft_consensus.cc:2272] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.500969 24780 ts_tablet_manager.cc:1916] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.476349 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44934f52cbeb448ba081b57cde783b2a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.509159 25492 log.cc:1199] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/72408899f710470baaf6fdb268cb04c8
I20251023 20:15:38.509603 25492 ts_tablet_manager.cc:1950] T 72408899f710470baaf6fdb268cb04c8 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:38.511229 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_2afbf83f519f4b7da2c4acc31ab18380 [id=6736a50ebad44c42985a5be788bfc43e]
I20251023 20:15:38.512048 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 401) took 3210 ms. Trace:
I20251023 20:15:38.512651 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_365d67170172442493eec0ae3918d5ff [id=6945386a3c8140c7b509b38c86326f5f]
I20251023 20:15:38.519950 23790 tablet_bootstrap.cc:492] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:38.520184 23790 ts_tablet_manager.cc:1403] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.001s
I20251023 20:15:38.521061 23790 raft_consensus.cc:359] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.521333 23790 raft_consensus.cc:385] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.521425 23790 raft_consensus.cc:740] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.521238 24780 ts_tablet_manager.cc:1929] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.521709 23790 consensus_queue.cc:260] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.521716 24780 log.cc:1199] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/925e53d9ea6f4c7baa38551eb917828d
I20251023 20:15:38.522229 23790 ts_tablet_manager.cc:1434] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:38.522357 24780 ts_tablet_manager.cc:1950] T 925e53d9ea6f4c7baa38551eb917828d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.522648 23790 tablet_bootstrap.cc:492] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.523190 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 183) took 9708 ms. Trace:
I20251023 20:15:38.523332 24780 rpcz_store.cc:276] 1023 20:15:28.815052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:28.815197 (+ 145us) service_pool.cc:225] Handling call
1023 20:15:38.499557 (+9684360us) ts_tablet_manager.cc:1151] Deleting tablet 925e53d9ea6f4c7baa38551eb917828d
1023 20:15:38.499567 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.511857 (+ 12290us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.521106 (+ 9249us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.523145 (+ 2039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9684226}
I20251023 20:15:38.523865 24780 tablet_replica.cc:333] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.524106 24780 raft_consensus.cc:2243] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.524868 24780 raft_consensus.cc:2272] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.526335 24780 ts_tablet_manager.cc:1916] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.524487 23790 tablet_bootstrap.cc:654] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.535384 24780 ts_tablet_manager.cc:1929] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251023 20:15:38.537268 24780 log.cc:1199] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/d2ae19cc80194363b8a9c96fbafb4ebe
I20251023 20:15:38.538215 24780 ts_tablet_manager.cc:1950] T d2ae19cc80194363b8a9c96fbafb4ebe P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.541239 23790 tablet_bootstrap.cc:492] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:38.541469 23790 ts_tablet_manager.cc:1403] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20251023 20:15:38.542184 23790 raft_consensus.cc:359] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.542454 23790 raft_consensus.cc:385] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.542580 23790 raft_consensus.cc:740] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.544086 23790 consensus_queue.cc:260] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.491017 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08b8c7c1b50d495f9ed54b8bfa1e984d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.501520 25784 raft_consensus.cc:493] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.544520 25784 raft_consensus.cc:515] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.544629 23333 leader_election.cc:304] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.477299 23786 tablet_bootstrap.cc:492] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.544881 25776 leader_election.cc:290] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.545035 23786 ts_tablet_manager.cc:1403] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.120s user 0.010s sys 0.000s
I20251023 20:15:38.545142 25784 leader_election.cc:290] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.545696 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c40164e687545c5b2d2751baaee4571" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.545687 23786 raft_consensus.cc:359] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.545841 23786 raft_consensus.cc:385] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.545722 25492 rpcz_store.cc:276] 1023 20:15:35.301413 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.302271 (+ 858us) service_pool.cc:225] Handling call
1023 20:15:38.389130 (+3086859us) ts_tablet_manager.cc:1151] Deleting tablet 72408899f710470baaf6fdb268cb04c8
1023 20:15:38.389140 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.431865 (+ 42725us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.473035 (+ 41170us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.512028 (+ 38993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3086641}
I20251023 20:15:38.545890 23786 raft_consensus.cc:740] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.546051 23786 consensus_queue.cc:260] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.546180 25492 tablet_replica.cc:333] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:38.546216 23471 leader_election.cc:343] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.546388 24821 raft_consensus.cc:3060] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.546423 23786 ts_tablet_manager.cc:1434] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:38.546689 23786 tablet_bootstrap.cc:492] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.547030 23790 ts_tablet_manager.cc:1434] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:38.547384 23790 tablet_bootstrap.cc:492] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.548081 23786 tablet_bootstrap.cc:654] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.543995 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 184) took 9466 ms. Trace:
I20251023 20:15:38.548498 24780 rpcz_store.cc:276] 1023 20:15:29.073223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:29.073286 (+ 63us) service_pool.cc:225] Handling call
1023 20:15:38.523784 (+9450498us) ts_tablet_manager.cc:1151] Deleting tablet d2ae19cc80194363b8a9c96fbafb4ebe
1023 20:15:38.523802 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.531768 (+ 7966us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.535255 (+ 3487us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.540106 (+ 4851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9450305}
I20251023 20:15:38.549239 24780 tablet_replica.cc:333] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.549526 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9749c6f3fe5545ab93bf49aae97e9818" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.550074 23605 leader_election.cc:343] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.550191 23605 leader_election.cc:304] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.550549 25776 raft_consensus.cc:2749] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.549525 24821 raft_consensus.cc:2749] T 08b8c7c1b50d495f9ed54b8bfa1e984d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.551652 24780 raft_consensus.cc:2243] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.552206 24821 consensus_queue.cc:1048] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.554901 24780 raft_consensus.cc:2272] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.555367 23661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c40164e687545c5b2d2751baaee4571" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.555531 23661 raft_consensus.cc:2468] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.555968 23470 leader_election.cc:304] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.556425 25784 raft_consensus.cc:2804] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.556504 25784 raft_consensus.cc:493] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.556558 25784 raft_consensus.cc:3060] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.559697 25784 raft_consensus.cc:515] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.560820 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c40164e687545c5b2d2751baaee4571" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:38.561404 23471 leader_election.cc:343] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.561765 25276 leader_election.cc:290] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.562240 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de2ead20b8fd4053af7cbc69cd247413" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.562733 23605 leader_election.cc:343] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.562934 23787 tablet_bootstrap.cc:492] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.563066 23787 ts_tablet_manager.cc:1403] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.159s user 0.010s sys 0.014s
I20251023 20:15:38.563284 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_21f538f7569b437b98fe8a5274592880 [id=6a431dc140754c84af27e9bd3ce01a12]
I20251023 20:15:38.563679 25784 leader_election.cc:290] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.563755 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c40164e687545c5b2d2751baaee4571" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.564222 24780 ts_tablet_manager.cc:1916] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.566672 23786 tablet_bootstrap.cc:492] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.566861 23786 ts_tablet_manager.cc:1403] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20251023 20:15:38.567744 25492 raft_consensus.cc:2243] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.563918 23666 raft_consensus.cc:3060] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.568728 25492 raft_consensus.cc:2272] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.568619 23786 raft_consensus.cc:359] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.569120 23786 raft_consensus.cc:385] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.569209 23786 raft_consensus.cc:740] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:38.569288 23470 consensus_peers.cc:597] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 370a790a053446b2b06314c7db242270. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:38.569521 23786 consensus_queue.cc:260] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.569747 24297 consensus_queue.cc:1048] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.570236 23786 ts_tablet_manager.cc:1434] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:38.571030 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_9cac41d47a584e59a3c3e6fd69f9ad79 [id=6b26d29083ab46a29f00f2552ef20892]
I20251023 20:15:38.571399 23786 tablet_bootstrap.cc:492] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.574772 23786 tablet_bootstrap.cc:654] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.581450 25492 ts_tablet_manager.cc:1916] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.584352 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641]
I20251023 20:15:38.584407 24297 consensus_queue.cc:1048] T c3d145a33b964a4ca1f0001ac6cf990b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.585925 23786 tablet_bootstrap.cc:492] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.586115 23786 ts_tablet_manager.cc:1403] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20251023 20:15:38.586876 23786 raft_consensus.cc:359] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.587182 23786 raft_consensus.cc:385] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.587270 23786 raft_consensus.cc:740] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.587507 23786 consensus_queue.cc:260] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.588066 23786 ts_tablet_manager.cc:1434] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:38.588374 23786 tablet_bootstrap.cc:492] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.592592 24780 ts_tablet_manager.cc:1929] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.567751 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.593142 23667 raft_consensus.cc:3060] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.593673 23786 tablet_bootstrap.cc:654] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.593663 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_f719c5f9fcda4036a855294384b2a635 [id=6c17b5f0dadf450ab07ff76ca1271bd3]
I20251023 20:15:38.594149 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 [id=6defcb8665844b1d958c71d4fc34263a]
I20251023 20:15:38.595723 24780 log.cc:1199] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/fb01b3f4ff7f41e98063ebf76abeace6
I20251023 20:15:38.596203 24780 ts_tablet_manager.cc:1950] T fb01b3f4ff7f41e98063ebf76abeace6 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.596375 25282 tablet_replica.cc:333] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:38.596591 25282 raft_consensus.cc:2243] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.596905 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de2ead20b8fd4053af7cbc69cd247413" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.602438 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 199) took 7605 ms. Trace:
I20251023 20:15:38.603013 24780 rpcz_store.cc:276] 1023 20:15:30.996579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.996631 (+ 52us) service_pool.cc:225] Handling call
1023 20:15:38.549178 (+7552547us) ts_tablet_manager.cc:1151] Deleting tablet fb01b3f4ff7f41e98063ebf76abeace6
1023 20:15:38.549187 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.578104 (+ 28917us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.592459 (+ 14355us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.602418 (+ 9959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7552366}
I20251023 20:15:38.603780 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 200) took 7607 ms. Trace:
I20251023 20:15:38.604120 24780 rpcz_store.cc:276] 1023 20:15:30.996671 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.996730 (+ 59us) service_pool.cc:225] Handling call
1023 20:15:38.603683 (+7606953us) ts_tablet_manager.cc:1151] Deleting tablet 7e08dad1d0f643efb3a821a1b9196290
1023 20:15:38.603693 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.603763 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7606783}
I20251023 20:15:38.604758 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 201) took 7608 ms. Trace:
I20251023 20:15:38.605106 24780 rpcz_store.cc:276] 1023 20:15:30.996760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.996938 (+ 178us) service_pool.cc:225] Handling call
1023 20:15:38.604682 (+7607744us) ts_tablet_manager.cc:1151] Deleting tablet ed46b7e67e4e4ae7b30ea69f1d3fdc2f
1023 20:15:38.604690 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.604743 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7607618}
I20251023 20:15:38.596760 25282 raft_consensus.cc:2272] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.605926 25282 ts_tablet_manager.cc:1916] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.599591 23667 raft_consensus.cc:2468] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.609957 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_cc8aa9734f8a4c83ab020f0e2ae90c12 [id=6eda60ed5ff344f3a610b8139aa5a6f5]
I20251023 20:15:38.610419 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_efda7d26e2124ec4b1eeca32fe1b545e [id=6f9ad1c9f64a44a4b856013bb8fd4bd1]
I20251023 20:15:38.610863 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_dce798c03c814b1899812dc8c06f4452 [id=6f9fc2e5ee6146e3a074b5fbd14314f1]
I20251023 20:15:38.611860 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a]
I20251023 20:15:38.613770 23666 raft_consensus.cc:2468] T 2c40164e687545c5b2d2751baaee4571 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.608446 25726 consensus_queue.cc:1048] T b3150a8478c144d285e999d5c9270fe0 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.614629 24780 tablet_replica.cc:333] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.617274 24780 raft_consensus.cc:2243] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 2 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.617413 23470 leader_election.cc:304] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.617688 24780 raft_consensus.cc:2272] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.617853 25784 raft_consensus.cc:2804] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.617970 25784 raft_consensus.cc:697] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.618172 25784 consensus_queue.cc:237] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.618474 24780 ts_tablet_manager.cc:1916] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.609344 23787 raft_consensus.cc:359] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.619287 23787 raft_consensus.cc:385] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.619351 23787 raft_consensus.cc:740] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.619531 23787 consensus_queue.cc:260] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.619982 23787 ts_tablet_manager.cc:1434] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20251023 20:15:38.620230 23787 tablet_bootstrap.cc:492] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.622473 23470 leader_election.cc:304] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.622891 25466 raft_consensus.cc:2804] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.623000 25466 raft_consensus.cc:697] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.623195 25466 consensus_queue.cc:237] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.624318 23790 tablet_bootstrap.cc:654] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.625787 24780 ts_tablet_manager.cc:1929] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251023 20:15:38.625921 24780 log.cc:1199] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/72408899f710470baaf6fdb268cb04c8
I20251023 20:15:38.626278 24780 ts_tablet_manager.cc:1950] T 72408899f710470baaf6fdb268cb04c8 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.626456 23787 tablet_bootstrap.cc:654] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.627331 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 203) took 7629 ms. Trace:
I20251023 20:15:38.628197 24780 rpcz_store.cc:276] 1023 20:15:30.997509 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.997566 (+ 57us) service_pool.cc:225] Handling call
1023 20:15:38.614578 (+7617012us) ts_tablet_manager.cc:1151] Deleting tablet 72408899f710470baaf6fdb268cb04c8
1023 20:15:38.614588 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.621295 (+ 6707us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.625704 (+ 4409us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.627310 (+ 1606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7616540}
I20251023 20:15:38.629456 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 206) took 7631 ms. Trace:
I20251023 20:15:38.629576 24780 rpcz_store.cc:276] 1023 20:15:30.997987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.998030 (+ 43us) service_pool.cc:225] Handling call
1023 20:15:38.628859 (+7630829us) ts_tablet_manager.cc:1151] Deleting tablet cb915d114fc0421999d079c23cad8eb5
1023 20:15:38.629355 (+ 496us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.629437 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7630676}
I20251023 20:15:38.629736 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_f5ea72074b174da28b3738378167f49c [id=70e0efabed5a4dfe98eb7201a2b4aba2]
I20251023 20:15:38.630007 24780 tablet_replica.cc:333] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.630679 24780 raft_consensus.cc:2243] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.631376 24780 raft_consensus.cc:2272] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.631814 23786 tablet_bootstrap.cc:492] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.631986 23786 ts_tablet_manager.cc:1403] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20251023 20:15:38.632308 24780 ts_tablet_manager.cc:1916] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.632513 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_5e48fffc45314e45a6f7ac8bba5ab31d [id=7260df2ce8034b73956131dd07502d7b]
I20251023 20:15:38.632763 23786 raft_consensus.cc:359] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.632987 23786 raft_consensus.cc:385] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.633072 23786 raft_consensus.cc:740] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.633327 23786 consensus_queue.cc:260] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.633849 23786 ts_tablet_manager.cc:1434] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:38.634137 23786 tablet_bootstrap.cc:492] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.635427 23786 tablet_bootstrap.cc:654] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.635648 25282 ts_tablet_manager.cc:1929] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.639060 24780 ts_tablet_manager.cc:1929] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.641291 23790 tablet_bootstrap.cc:492] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:38.641588 25492 ts_tablet_manager.cc:1929] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.641734 23790 ts_tablet_manager.cc:1403] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.094s user 0.019s sys 0.000s
I20251023 20:15:38.642393 23790 raft_consensus.cc:359] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.642530 23790 raft_consensus.cc:385] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.642586 23790 raft_consensus.cc:740] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.642745 23790 consensus_queue.cc:260] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.643137 23790 ts_tablet_manager.cc:1434] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251023 20:15:38.643368 23790 tablet_bootstrap.cc:492] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.643702 25282 log.cc:1199] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/377de4ea0bd6489284fac6b0fab52c8c
I20251023 20:15:38.644590 23387 raft_consensus.cc:3060] T 3befdfc85bca42bb816e11d1dd832c8c P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.645175 25282 ts_tablet_manager.cc:1950] T 377de4ea0bd6489284fac6b0fab52c8c P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:38.646467 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_572a1f2b4ae0442ba59eb601fffb021c [id=753d0b7ee3444bb9bfe444b63a1ac511]
I20251023 20:15:38.647035 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 383) took 5090 ms. Trace:
I20251023 20:15:38.647127 25282 rpcz_store.cc:276] 1023 20:15:33.556895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.557298 (+ 403us) service_pool.cc:225] Handling call
1023 20:15:38.449858 (+4892560us) ts_tablet_manager.cc:1151] Deleting tablet 377de4ea0bd6489284fac6b0fab52c8c
1023 20:15:38.568411 (+118553us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x61800000d518. stack: 00007fb96b1373e6 00007fb96b1371a0 00007fb96a11529d 0000000000983974 000000000098393d 000000000098330a 00007fb96b11fced 00007fb96b120d5e 00007fb97ed3b1dd 00007fb97ecb9743 00007fb97aeed1c6 00007fb97aeee549 00007fb97af21e54 00007fb97af028f0 00007fb97af0267d 0000000000a880ec
1023 20:15:38.596327 (+ 27916us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.616729 (+ 20402us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.635541 (+ 18812us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.647012 (+ 11471us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31205248,"tablet-delete.queue_time_us":4892388}
I20251023 20:15:38.647811 25466 consensus_queue.cc:1048] T 3befdfc85bca42bb816e11d1dd832c8c P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.648098 23790 tablet_bootstrap.cc:654] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.641680 25492 log.cc:1199] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/fb01b3f4ff7f41e98063ebf76abeace6
I20251023 20:15:38.651057 25492 ts_tablet_manager.cc:1950] T fb01b3f4ff7f41e98063ebf76abeace6 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:38.654102 25282 tablet_replica.cc:333] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:38.656863 25282 raft_consensus.cc:2243] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.657948 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_a5e06ba2ba784868978a86ee8e607a2e [id=7718e9dc72d5495d8c9cb61bac6caf44]
I20251023 20:15:38.658358 23786 tablet_bootstrap.cc:492] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.658492 23786 ts_tablet_manager.cc:1403] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251023 20:15:38.659046 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_2f774f0ffb2e4453a2ccbf6ed961a1e6 [id=77ff0c7e54e94defabffdbc82dbb609d]
I20251023 20:15:38.659205 23786 raft_consensus.cc:359] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.659377 23786 raft_consensus.cc:385] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.659420 23786 raft_consensus.cc:740] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.659613 23786 consensus_queue.cc:260] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.660066 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3ad1b2da425540ccbe8b55d22f379b4a [id=7810e4d5193c4559891f1603fe361e11]
I20251023 20:15:38.660082 23786 ts_tablet_manager.cc:1434] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251023 20:15:38.660336 23786 tablet_bootstrap.cc:492] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.660465 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_a9bddf9d668f414dae97b08900de3f65 [id=78aea9dba8d94e5ca65f8c397f78395c]
I20251023 20:15:38.660842 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d22911da756748fb8939006df660e885 [id=78fe983663c94eabab93ee902bde6998]
I20251023 20:15:38.661891 23786 tablet_bootstrap.cc:654] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.663763 25282 raft_consensus.cc:2272] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:38.667405 23604 leader_election.cc:343] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.668421 23786 tablet_bootstrap.cc:492] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.668658 23786 ts_tablet_manager.cc:1403] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20251023 20:15:38.669432 24780 log.cc:1199] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/05933cb9433d48d285b2db82b289bdc8
I20251023 20:15:38.669433 23786 raft_consensus.cc:359] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.669736 23786 raft_consensus.cc:385] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.669802 23786 raft_consensus.cc:740] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.670037 23786 consensus_queue.cc:260] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.670554 23786 ts_tablet_manager.cc:1434] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:38.670740 25167 raft_consensus.cc:493] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.670907 23786 tablet_bootstrap.cc:492] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.670855 25167 raft_consensus.cc:515] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.670960 24780 ts_tablet_manager.cc:1950] T 05933cb9433d48d285b2db82b289bdc8 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.671121 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 [id=79163b61e42a43d8879399cd5f31ce44]
I20251023 20:15:38.671388 25167 leader_election.cc:290] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.671548 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_f021ca39359245598311be2bc68bbbcc [id=7a4098cc9d4e49a39ef792883f85d564]
I20251023 20:15:38.671777 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a45d9b3031b46d390e0a1010489c718" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.672247 23471 leader_election.cc:343] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.672601 23786 tablet_bootstrap.cc:654] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.670657 23604 leader_election.cc:304] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.673839 25854 raft_consensus.cc:2749] T 44934f52cbeb448ba081b57cde783b2a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.674062 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 205) took 7676 ms. Trace:
I20251023 20:15:38.675207 24780 rpcz_store.cc:276] 1023 20:15:30.997886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.997988 (+ 102us) service_pool.cc:225] Handling call
1023 20:15:38.629964 (+7631976us) ts_tablet_manager.cc:1151] Deleting tablet 05933cb9433d48d285b2db82b289bdc8
1023 20:15:38.629972 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.636226 (+ 6254us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.638972 (+ 2746us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.674042 (+ 35070us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7630934}
I20251023 20:15:38.675400 25282 ts_tablet_manager.cc:1916] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.676190 24780 tablet_replica.cc:333] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.676723 24780 raft_consensus.cc:2243] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
W20251023 20:15:38.676806 23604 leader_election.cc:343] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.677518 24780 raft_consensus.cc:2272] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.678653 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_34d8a4d24be94b1ea676b7a0df81b96c [id=7b824b39feed48148896732770542d20]
I20251023 20:15:38.678869 24780 ts_tablet_manager.cc:1916] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.676998 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a45d9b3031b46d390e0a1010489c718" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.679826 25854 raft_consensus.cc:493] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.681155 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48]
I20251023 20:15:38.682389 25854 raft_consensus.cc:515] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.683423 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_a5331f0f356d4ed88cf6787eb94db083 [id=7bf105aabdc944afac7c5d1e5afb824b]
I20251023 20:15:38.683446 25854 leader_election.cc:290] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.682479 23668 raft_consensus.cc:2468] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.682374 23604 leader_election.cc:304] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.684223 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf858e0fc0634ef3868656d5dea6c82a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.684877 23605 leader_election.cc:343] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.685029 25854 raft_consensus.cc:2749] T de2ead20b8fd4053af7cbc69cd247413 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.685173 23470 leader_election.cc:304] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.685416 25466 raft_consensus.cc:2804] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.685489 25466 raft_consensus.cc:493] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.685540 25466 raft_consensus.cc:3060] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.687919 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_c462519a21af4c17bc119f202f50922a [id=7fb99f9e0faa4759ad4c12b5ce1767ea]
I20251023 20:15:38.688056 25466 raft_consensus.cc:515] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.691023 23786 tablet_bootstrap.cc:492] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.691147 23786 ts_tablet_manager.cc:1403] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20251023 20:15:38.691776 23786 raft_consensus.cc:359] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.691929 23786 raft_consensus.cc:385] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.691974 23786 raft_consensus.cc:740] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.692128 23786 consensus_queue.cc:260] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.692567 23786 ts_tablet_manager.cc:1434] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251023 20:15:38.694264 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_775428d3af964d538226c836574a899d [id=818f0fca06904f09ad46c9e184c982b4]
I20251023 20:15:38.694721 25466 leader_election.cc:290] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.695052 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a45d9b3031b46d390e0a1010489c718" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
W20251023 20:15:38.696120 23471 leader_election.cc:343] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.700317 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a45d9b3031b46d390e0a1010489c718" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.700615 23668 raft_consensus.cc:3060] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.701645 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7]
I20251023 20:15:38.702240 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3eeb50ae897f4006ad63033f2ad0324a [id=8477c10fb2f4422996d4eab8717d7fe8]
I20251023 20:15:38.702862 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477]
I20251023 20:15:38.703526 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_c1f42ba6bf654a09ade937991a5f85c0 [id=85e0bb2abec640a28b1cc1806b45e52a]
I20251023 20:15:38.703845 23668 raft_consensus.cc:2468] T 8a45d9b3031b46d390e0a1010489c718 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.704164 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ef7d0ab458b247c5a1998f0538a7b2ea [id=87bf96dc0f3b4ca6867bb468b7501138]
I20251023 20:15:38.708590 25822 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6545140d94b7485aa81c51ad233355cd)
I20251023 20:15:38.708788 25822 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6545140d94b7485aa81c51ad233355cd" member_type: VOTER last_known_addr { host: "127.22.127.190" port: 34331 } } peers { permanent_uuid: "ae5a310b4a514c0f812bda133c756062" member_type: VOTER last_known_addr { host: "127.22.127.189" port: 33509 } } peers { permanent_uuid: "690ffde75cd745b6a87925967d54471a" member_type: VOTER last_known_addr { host: "127.22.127.188" port: 41417 } }
I20251023 20:15:38.709563 25822 leader_election.cc:290] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331), 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417)
I20251023 20:15:38.709122 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 404) took 3407 ms. Trace:
I20251023 20:15:38.711100 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf858e0fc0634ef3868656d5dea6c82a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.709924 25492 rpcz_store.cc:276] 1023 20:15:35.301738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.302242 (+ 504us) service_pool.cc:225] Handling call
1023 20:15:38.546135 (+3243893us) ts_tablet_manager.cc:1151] Deleting tablet fb01b3f4ff7f41e98063ebf76abeace6
1023 20:15:38.546144 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.625344 (+ 79200us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.641489 (+ 16145us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.708863 (+ 67374us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53597,"tablet-delete.queue_time_us":3243550}
I20251023 20:15:38.713371 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 406) took 3411 ms. Trace:
I20251023 20:15:38.708104 23470 leader_election.cc:304] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.714663 25466 raft_consensus.cc:2804] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.714809 25466 raft_consensus.cc:697] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.715049 25466 consensus_queue.cc:237] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:38.715751 23603 connection.cc:537] server connection from 127.0.0.1:60202 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:38.715781 23604 leader_election.cc:343] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.715898 23604 leader_election.cc:304] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.716115 25854 raft_consensus.cc:2749] T bf858e0fc0634ef3868656d5dea6c82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.716439 23949 connection.cc:537] client connection to 127.22.127.188:41417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:38.716557 23949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:38.716164 23469 connection.cc:537] server connection from 127.0.0.1:48312 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:38.717568 23786 tablet_bootstrap.cc:492] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.720042 23787 tablet_bootstrap.cc:492] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.720168 23787 ts_tablet_manager.cc:1403] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.005s
I20251023 20:15:38.720906 23787 raft_consensus.cc:359] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.721071 23787 raft_consensus.cc:385] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.721149 23787 raft_consensus.cc:740] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.721457 23787 consensus_queue.cc:260] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.722092 23787 ts_tablet_manager.cc:1434] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:38.722354 23787 tablet_bootstrap.cc:492] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.723883 23787 tablet_bootstrap.cc:654] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:38.730695 23722 connection.cc:537] client connection to 127.22.127.188:41417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:38.731848 23722 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.22.127.188:41417)
W20251023 20:15:38.732396 23722 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.127.188:41417)
I20251023 20:15:38.732903 23786 tablet_bootstrap.cc:654] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:38.733065 23722 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.22.127.188:41417)
W20251023 20:15:38.733848 23722 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.22.127.188:41417)
I20251023 20:15:38.738682 23790 tablet_bootstrap.cc:492] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:38.739223 23790 ts_tablet_manager.cc:1403] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.004s
I20251023 20:15:38.742306 23790 raft_consensus.cc:359] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.742529 23790 raft_consensus.cc:385] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.744892 23790 raft_consensus.cc:740] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.745241 23790 consensus_queue.cc:260] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.750154 25854 raft_consensus.cc:493] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.750254 23790 ts_tablet_manager.cc:1434] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251023 20:15:38.750286 25854 raft_consensus.cc:515] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.750543 23790 tablet_bootstrap.cc:492] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.750850 25854 leader_election.cc:290] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.752447 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4023a5776334f42a2564f538a7c5fee" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.752636 24780 ts_tablet_manager.cc:1929] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.752728 24780 log.cc:1199] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/514cb3b1ed024041ac160a7d9acff618
W20251023 20:15:38.752775 23605 leader_election.cc:343] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: f4023a5776334f42a2564f538a7c5fee
W20251023 20:15:38.753145 23604 leader_election.cc:343] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.753276 23604 leader_election.cc:304] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.753530 25854 raft_consensus.cc:2749] T f4023a5776334f42a2564f538a7c5fee P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.753885 24780 ts_tablet_manager.cc:1950] T 514cb3b1ed024041ac160a7d9acff618 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.755980 25282 ts_tablet_manager.cc:1929] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.755805 25784 consensus_queue.cc:1048] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.756507 25282 log.cc:1199] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/d62dbf044acd4459b26d2e7e50f6206e
I20251023 20:15:38.757795 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 198) took 7761 ms. Trace:
I20251023 20:15:38.758508 23787 tablet_bootstrap.cc:492] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.758648 23787 ts_tablet_manager.cc:1403] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20251023 20:15:38.759146 24780 rpcz_store.cc:276] 1023 20:15:30.996472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.996529 (+ 57us) service_pool.cc:225] Handling call
1023 20:15:38.676144 (+7679615us) ts_tablet_manager.cc:1151] Deleting tablet 514cb3b1ed024041ac160a7d9acff618
1023 20:15:38.676154 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.736917 (+ 60763us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.752521 (+ 15604us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.757774 (+ 5253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7675443}
I20251023 20:15:38.759402 23787 raft_consensus.cc:359] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.759586 23787 raft_consensus.cc:385] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.753010 23790 tablet_bootstrap.cc:654] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.759742 23787 raft_consensus.cc:740] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.760165 23787 consensus_queue.cc:260] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.762064 23786 tablet_bootstrap.cc:492] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.762243 23786 ts_tablet_manager.cc:1403] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20251023 20:15:38.762912 23786 raft_consensus.cc:359] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.763228 23786 raft_consensus.cc:385] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.763315 23786 raft_consensus.cc:740] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.763394 24780 tablet_replica.cc:333] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.763592 23786 consensus_queue.cc:260] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.764101 23786 ts_tablet_manager.cc:1434] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:38.764381 23786 tablet_bootstrap.cc:492] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.764138 24780 raft_consensus.cc:2243] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:38.765051 24780 raft_consensus.cc:2272] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.765908 23786 tablet_bootstrap.cc:654] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.766332 24780 ts_tablet_manager.cc:1916] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:38.767764 23603 connection.cc:537] client connection to 127.22.127.188:41417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:38.770045 23790 tablet_bootstrap.cc:492] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:38.773491 25739 consensus_queue.cc:1048] T a1efa4fcaeb049b484ca59b12025ce9a P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.773892 23787 ts_tablet_manager.cc:1434] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20251023 20:15:38.776202 25739 consensus_queue.cc:1048] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.777920 23469 connection.cc:537] client connection to 127.22.127.188:41417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251023 20:15:38.780939 24840 connection.cc:537] server connection from 127.0.0.1:52056 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:38.770198 23790 ts_tablet_manager.cc:1403] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20251023 20:15:38.786628 23790 raft_consensus.cc:359] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.789165 23790 raft_consensus.cc:385] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.789817 24297 consensus_queue.cc:1048] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.789395 23790 raft_consensus.cc:740] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.790887 23790 consensus_queue.cc:260] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.791872 23790 ts_tablet_manager.cc:1434] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:38.794127 23787 tablet_bootstrap.cc:492] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.795465 23790 tablet_bootstrap.cc:492] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.796144 25739 consensus_queue.cc:1048] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.797056 25282 ts_tablet_manager.cc:1950] T d62dbf044acd4459b26d2e7e50f6206e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:38.797959 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 384) took 5240 ms. Trace:
I20251023 20:15:38.798108 25282 rpcz_store.cc:276] 1023 20:15:33.557963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.558024 (+ 61us) service_pool.cc:225] Handling call
1023 20:15:38.653451 (+5095427us) ts_tablet_manager.cc:1151] Deleting tablet d62dbf044acd4459b26d2e7e50f6206e
1023 20:15:38.653462 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.692525 (+ 39063us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.754834 (+ 62309us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.797939 (+ 43105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5094826}
I20251023 20:15:38.798817 25282 tablet_replica.cc:333] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:38.801817 23786 tablet_bootstrap.cc:492] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.802011 23786 ts_tablet_manager.cc:1403] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.003s
I20251023 20:15:38.802713 23786 raft_consensus.cc:359] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.802943 23786 raft_consensus.cc:385] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.803025 23786 raft_consensus.cc:740] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.803314 23786 consensus_queue.cc:260] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.803996 23786 ts_tablet_manager.cc:1434] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:38.805157 23786 tablet_bootstrap.cc:492] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.806763 23786 tablet_bootstrap.cc:654] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.813776 25282 raft_consensus.cc:2243] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.814179 25282 raft_consensus.cc:2272] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.815363 25282 ts_tablet_manager.cc:1916] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.815985 24780 ts_tablet_manager.cc:1929] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.820298 24780 log.cc:1199] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/c9bfbf0acdc84f2f8fe553694ee9897c
W20251023 20:15:38.820614 24831 connection.cc:537] client connection to 127.22.127.188:41417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:38.820971 24780 ts_tablet_manager.cc:1950] T c9bfbf0acdc84f2f8fe553694ee9897c P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:38.820894 24831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:38.817740 23790 tablet_bootstrap.cc:654] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:38.820515 23333 connection.cc:537] server connection from 127.0.0.1:38604 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251023 20:15:38.826208 25167 consensus_queue.cc:1048] T fa0a46ffa626447eab4cdb956e798005 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.826349 25276 raft_consensus.cc:493] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.826481 25276 raft_consensus.cc:515] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.827072 25276 leader_election.cc:290] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.828008 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea3149f906064f8c8167292ec86540fd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.828225 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea3149f906064f8c8167292ec86540fd" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.828581 23604 leader_election.cc:343] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:38.828953 25887 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:15:38.784941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.127.188:41417 (local address 127.22.127.129:55391)
1023 20:15:38.804451 (+ 19510us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:38.804474 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1023 20:15:38.804632 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1023 20:15:38.828708 (+ 24076us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":19379,"thread_start_us":19780,"threads_started":1}
W20251023 20:15:38.830705 23605 leader_election.cc:343] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.830832 23605 leader_election.cc:304] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.831224 25276 raft_consensus.cc:2749] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.824551 23787 tablet_bootstrap.cc:654] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.833709 24297 consensus_queue.cc:1048] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.833950 25167 consensus_queue.cc:1048] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.835052 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_c760c475c8554c89aeb6fd77fd422797 [id=886ab40a90ab4410aa13fabf17a1ea91]
I20251023 20:15:38.836467 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3f23d19f3e734d84b7c5a1e0a0d83726 [id=89fe830110bb49bd891eb42bac7215d7]
I20251023 20:15:38.837337 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3f6d1412904e4106b793f17c29014aec [id=8a388a26b69d416ab99d15c024f1d3f3]
I20251023 20:15:38.838052 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_0984ad1922884cb19d48dc21a903dfe2 [id=8b419bc021804ce7ab372b0515b672f2]
I20251023 20:15:38.839792 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_014b7052f2aa423e8f841e4be8da3a9c [id=8d6a76224aec482db23a4b660e13a98c]
I20251023 20:15:38.840210 25167 raft_consensus.cc:493] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.840340 25167 raft_consensus.cc:515] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.840950 25167 leader_election.cc:290] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.841497 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:38.842381 23471 leader_election.cc:343] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.841989 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.842931 23669 raft_consensus.cc:2468] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.843326 23470 leader_election.cc:304] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.843550 25167 raft_consensus.cc:2804] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.843631 25167 raft_consensus.cc:493] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.843694 25167 raft_consensus.cc:3060] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.846228 25167 raft_consensus.cc:515] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.847306 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.847412 25167 leader_election.cc:290] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:38.847764 23471 leader_election.cc:343] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.847512 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.848153 23669 raft_consensus.cc:3060] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.849004 23787 tablet_bootstrap.cc:492] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:38.850180 23787 ts_tablet_manager.cc:1403] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.000s
W20251023 20:15:38.850679 23333 consensus_peers.cc:597] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db758a7a7a89410f8ac1da72048fbc56. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:38.851014 23790 tablet_bootstrap.cc:492] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:38.851202 23790 ts_tablet_manager.cc:1403] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.002s
I20251023 20:15:38.851890 23790 raft_consensus.cc:359] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.852208 23790 raft_consensus.cc:385] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.852351 23787 raft_consensus.cc:359] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.852440 23790 raft_consensus.cc:740] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.852828 23787 raft_consensus.cc:385] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.852905 23787 raft_consensus.cc:740] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.853195 23787 consensus_queue.cc:260] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.853703 23787 ts_tablet_manager.cc:1434] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251023 20:15:38.854214 23787 tablet_bootstrap.cc:492] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:38.856734 23787 tablet_bootstrap.cc:654] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.865929 25167 consensus_queue.cc:1048] T d227b45bfcc64d76a51cff347e6e1486 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.853015 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 204) took 7855 ms. Trace:
I20251023 20:15:38.856582 25594 consensus_queue.cc:1048] T 225ce6aa52c54f76bd56247f01219dbc P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.858107 24297 consensus_queue.cc:1048] T 60e8942deb7544a09b21c3d1c2a3f99f P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.858506 25726 consensus_queue.cc:1048] T db758a7a7a89410f8ac1da72048fbc56 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.871376 24780 rpcz_store.cc:276] 1023 20:15:30.997628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.997666 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:38.763345 (+7765679us) ts_tablet_manager.cc:1151] Deleting tablet c9bfbf0acdc84f2f8fe553694ee9897c
1023 20:15:38.763355 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.797644 (+ 34289us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.815862 (+ 18218us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.852968 (+ 37106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7762512}
I20251023 20:15:38.871577 25854 raft_consensus.cc:493] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.871695 25854 raft_consensus.cc:515] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.872229 25854 leader_election.cc:290] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.872717 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d801ca6ce63440abefe0c694889fa9e" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:38.873325 23604 leader_election.cc:343] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:38.874132 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 204) could send its response
W20251023 20:15:38.879213 25886 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:15:38.781613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.127.188:41417 (local address 127.22.127.130:48677)
1023 20:15:38.878875 (+ 97262us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:38.878967 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":97162,"threads_started":1}
I20251023 20:15:38.880295 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 202) took 7883 ms. Trace:
I20251023 20:15:38.880475 24780 rpcz_store.cc:276] 1023 20:15:30.996836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.996915 (+ 79us) service_pool.cc:225] Handling call
1023 20:15:38.871761 (+7874846us) ts_tablet_manager.cc:1151] Deleting tablet f1a6fe726ef844829a7ffec9d4494637
1023 20:15:38.880190 (+ 8429us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.880275 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7850322}
W20251023 20:15:38.880849 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 202) could send its response
W20251023 20:15:38.881444 23605 leader_election.cc:343] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Not found: Tablet not found: 5d801ca6ce63440abefe0c694889fa9e
I20251023 20:15:38.881573 23605 leader_election.cc:304] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
W20251023 20:15:38.881822 25883 negotiation.cc:337] Failed RPC negotiation. Trace:
1023 20:15:38.771465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.127.188:41417 (local address 127.22.127.131:47197)
1023 20:15:38.881370 (+109905us) negotiation.cc:107] Waiting for socket to connect
1023 20:15:38.881487 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":109812,"threads_started":1}
I20251023 20:15:38.881968 25276 raft_consensus.cc:2749] T 5d801ca6ce63440abefe0c694889fa9e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:38.882226 24780 tablet_replica.cc:333] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.882412 24780 raft_consensus.cc:2243] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.882573 24780 raft_consensus.cc:2272] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.883116 24780 ts_tablet_manager.cc:1916] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.885205 23790 consensus_queue.cc:260] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.886025 23790 ts_tablet_manager.cc:1434] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20251023 20:15:38.886279 23790 tablet_bootstrap.cc:492] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:38.887205 25739 raft_consensus.cc:493] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.887329 25739 raft_consensus.cc:515] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.887874 25739 leader_election.cc:290] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.888715 23786 tablet_bootstrap.cc:492] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.888870 23786 ts_tablet_manager.cc:1403] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.009s
I20251023 20:15:38.889528 23786 raft_consensus.cc:359] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.889694 23786 raft_consensus.cc:385] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.889803 23786 raft_consensus.cc:740] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
W20251023 20:15:38.889781 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 384) could send its response
I20251023 20:15:38.890055 23786 consensus_queue.cc:260] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.890252 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ee1974eb5e343b7953dadda7d5799b1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.890427 23663 raft_consensus.cc:2468] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:38.890579 23786 ts_tablet_manager.cc:1434] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:38.890914 23786 tablet_bootstrap.cc:492] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.891032 23470 leader_election.cc:304] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.891350 25784 raft_consensus.cc:2804] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.891412 25784 raft_consensus.cc:493] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.891464 25784 raft_consensus.cc:3060] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.892843 23790 tablet_bootstrap.cc:654] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.894150 25784 raft_consensus.cc:515] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.894625 25784 leader_election.cc:290] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.895488 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ee1974eb5e343b7953dadda7d5799b1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.895639 23663 raft_consensus.cc:3060] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.896518 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ee1974eb5e343b7953dadda7d5799b1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:38.713940 25492 rpcz_store.cc:276] 1023 20:15:35.301933 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.302469 (+ 536us) service_pool.cc:225] Handling call
1023 20:15:38.712152 (+3409683us) ts_tablet_manager.cc:1151] Deleting tablet 48436f3d8f1443db990a2d10b668b70a
1023 20:15:38.712507 (+ 355us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.713025 (+ 518us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3409545}
I20251023 20:15:38.897022 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 409) took 3594 ms. Trace:
I20251023 20:15:38.897068 25726 raft_consensus.cc:493] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.897109 25492 rpcz_store.cc:276] 1023 20:15:35.302213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.302606 (+ 393us) service_pool.cc:225] Handling call
1023 20:15:38.896935 (+3594329us) ts_tablet_manager.cc:1151] Deleting tablet 352812b56f8d4feb818a364d6adfd86d
1023 20:15:38.896944 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.897006 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3594214}
I20251023 20:15:38.897200 25726 raft_consensus.cc:515] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.897359 25492 tablet_replica.cc:333] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:38.897516 25492 raft_consensus.cc:2243] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.897660 25492 raft_consensus.cc:2272] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.897680 25726 leader_election.cc:290] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.898241 23663 raft_consensus.cc:2468] T 6ee1974eb5e343b7953dadda7d5799b1 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:38.898553 23661 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.898725 23661 raft_consensus.cc:2393] T a841ffdc6cdd432dac7afe2e2b53f224 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:38.899129 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ee1974eb5e343b7953dadda7d5799b1" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.899302 23470 leader_election.cc:304] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:38.899874 23471 leader_election.cc:343] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.899890 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a841ffdc6cdd432dac7afe2e2b53f224" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.900694 25167 raft_consensus.cc:2804] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.900825 25167 raft_consensus.cc:697] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.901121 25167 consensus_queue.cc:237] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.902515 23333 leader_election.cc:304] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.903095 25592 raft_consensus.cc:3060] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:38.903654 23471 leader_election.cc:343] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.905441 23786 tablet_bootstrap.cc:654] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.905601 25592 raft_consensus.cc:2749] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.908322 24831 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.188:41417, user_credentials={real_user=slave}} blocked reactor thread for 87515.2us
I20251023 20:15:38.913156 25776 raft_consensus.cc:493] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.913290 25776 raft_consensus.cc:515] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:38.913879 25776 leader_election.cc:290] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:38.914281 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68f5589f890f47e286a70419c2bf57c5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:38.914497 25167 consensus_queue.cc:1048] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.914673 23605 leader_election.cc:343] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.908533 24423 master_runner.cc:186] Error getting master registration for 127.22.127.188:41417: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20251023 20:15:38.915548 24423 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251023 20:15:38.917199 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68f5589f890f47e286a70419c2bf57c5" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:38.917645 23604 leader_election.cc:343] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:38.918751 23604 leader_election.cc:304] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:38.919209 25854 raft_consensus.cc:2749] T 68f5589f890f47e286a70419c2bf57c5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:38.920933 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 406) could send its response
I20251023 20:15:38.923027 25167 consensus_queue.cc:1048] T a4d71f0cfdb6449fbd74b854c15326d6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.916925 23669 raft_consensus.cc:2468] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
W20251023 20:15:38.925963 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 409) could send its response
I20251023 20:15:38.926551 23470 leader_election.cc:304] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:38.926987 25784 raft_consensus.cc:2804] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.927219 25784 raft_consensus.cc:697] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:38.927606 25784 consensus_queue.cc:237] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.929781 24780 ts_tablet_manager.cc:1929] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:38.929909 25167 consensus_queue.cc:1048] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.931202 24780 log.cc:1199] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/9ed0aa7e36e64843ad6b2ad4b415832a
I20251023 20:15:38.931718 24780 ts_tablet_manager.cc:1950] T 9ed0aa7e36e64843ad6b2ad4b415832a P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:38.933806 25167 consensus_queue.cc:1048] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:38.934779 25492 ts_tablet_manager.cc:1916] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.937018 25726 raft_consensus.cc:493] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:38.937208 25726 raft_consensus.cc:515] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.937759 25726 leader_election.cc:290] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.938520 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31df5c2c40de4d88b6ab1567b84ce4ef" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:38.938709 23525 raft_consensus.cc:2468] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:38.938637 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31df5c2c40de4d88b6ab1567b84ce4ef" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:38.939510 23669 raft_consensus.cc:2468] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:38.940070 23333 leader_election.cc:304] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:38.940449 25726 raft_consensus.cc:2804] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:38.940567 25726 raft_consensus.cc:493] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:38.940678 25726 raft_consensus.cc:3060] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.943295 25726 raft_consensus.cc:515] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.943919 25726 leader_election.cc:290] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:38.945204 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31df5c2c40de4d88b6ab1567b84ce4ef" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:38.945349 23525 raft_consensus.cc:3060] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.948140 23525 raft_consensus.cc:2468] T 31df5c2c40de4d88b6ab1567b84ce4ef P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:38.948462 23786 tablet_bootstrap.cc:492] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.948663 23786 ts_tablet_manager.cc:1403] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.000s
I20251023 20:15:38.952001 23786 raft_consensus.cc:359] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.954490 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_39ef492038e649d1b5378fedd9cda522 [id=8ddeeaea0bf548eab7bef76cd980a311]
I20251023 20:15:38.955367 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 207) took 7957 ms. Trace:
I20251023 20:15:38.955420 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_2dc2a5b423424735a1c370afde321275 [id=8f71d213abd24c078ff9db0f0b20c301]
I20251023 20:15:38.956033 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ac0a68d0e9b7477892d3a8e14a3d7f4b [id=905ffd6fc69a44c39b12a7efecc70fd4]
I20251023 20:15:38.956436 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6]
I20251023 20:15:38.956575 24780 rpcz_store.cc:276] 1023 20:15:30.998388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:30.998428 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:38.880867 (+7882439us) ts_tablet_manager.cc:1151] Deleting tablet 9ed0aa7e36e64843ad6b2ad4b415832a
1023 20:15:38.882184 (+ 1317us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.907810 (+ 25626us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.929662 (+ 21852us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:38.955345 (+ 25683us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40,"tablet-delete.queue_time_us":7859399}
W20251023 20:15:38.893136 23949 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.188:41417, user_credentials={real_user=slave}} blocked reactor thread for 168498us
I20251023 20:15:38.957557 23786 raft_consensus.cc:385] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.957580 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 210) took 7923 ms. Trace:
I20251023 20:15:38.957664 24780 rpcz_store.cc:276] 1023 20:15:31.033791 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.033992 (+ 201us) service_pool.cc:225] Handling call
1023 20:15:38.957469 (+7923477us) ts_tablet_manager.cc:1151] Deleting tablet aa794b3bec4b4b3993c814ea7cd02ee2
1023 20:15:38.957480 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.957563 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7923293}
W20251023 20:15:38.957774 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 207) could send its response
I20251023 20:15:38.957973 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 211) took 7924 ms. Trace:
I20251023 20:15:38.958081 23333 leader_election.cc:304] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:38.957986 23786 raft_consensus.cc:740] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.958534 24297 raft_consensus.cc:2804] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:38.958678 24297 raft_consensus.cc:697] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Becoming Leader. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Running, Role: LEADER
W20251023 20:15:38.958693 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 210) could send its response
I20251023 20:15:38.958644 23786 consensus_queue.cc:260] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.958894 24297 consensus_queue.cc:237] T 31df5c2c40de4d88b6ab1567b84ce4ef P d220aba7c9c0400398ffde917376ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.959900 23398 raft_consensus.cc:3060] T 59598ea40b9e4e5da0ec36f35f1094d6 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.959923 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "31df5c2c40de4d88b6ab1567b84ce4ef" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:38.960099 23668 raft_consensus.cc:3060] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:38.958047 24780 rpcz_store.cc:276] 1023 20:15:31.033861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034202 (+ 341us) service_pool.cc:225] Handling call
1023 20:15:38.957897 (+7923695us) ts_tablet_manager.cc:1151] Deleting tablet 48436f3d8f1443db990a2d10b668b70a
1023 20:15:38.957906 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.957959 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7923571}
I20251023 20:15:38.960665 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 213) took 7926 ms. Trace:
I20251023 20:15:38.960726 24780 rpcz_store.cc:276] 1023 20:15:31.034014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034350 (+ 336us) service_pool.cc:225] Handling call
1023 20:15:38.960593 (+7926243us) ts_tablet_manager.cc:1151] Deleting tablet 352812b56f8d4feb818a364d6adfd86d
1023 20:15:38.960601 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.960652 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7926120}
I20251023 20:15:38.961619 23786 ts_tablet_manager.cc:1434] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251023 20:15:38.961767 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 220) took 7927 ms. Trace:
I20251023 20:15:38.961879 23786 tablet_bootstrap.cc:492] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.961843 24780 rpcz_store.cc:276] 1023 20:15:31.034614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034854 (+ 240us) service_pool.cc:225] Handling call
1023 20:15:38.961697 (+7926843us) ts_tablet_manager.cc:1151] Deleting tablet ef1229b9e0384a82811f244d2bf4c8ce
1023 20:15:38.961705 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.961755 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7925293}
I20251023 20:15:38.962040 24780 tablet_replica.cc:333] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:38.962213 24780 raft_consensus.cc:2243] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:38.962332 24780 raft_consensus.cc:2272] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:38.962852 24780 ts_tablet_manager.cc:1916] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:38.963223 23786 tablet_bootstrap.cc:654] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:38.964284 23470 consensus_peers.cc:597] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbab01ecb27a426188689408079eaa01. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:38.964001 25167 consensus_queue.cc:1048] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.962044 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 211) could send its response
W20251023 20:15:38.965579 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 213) could send its response
I20251023 20:15:38.966223 24297 consensus_queue.cc:1048] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.966250 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 220) could send its response
I20251023 20:15:38.966672 25784 consensus_queue.cc:1048] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:38.970404 24831 leader_election.cc:336] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417): Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111)
I20251023 20:15:38.977586 23668 raft_consensus.cc:2468] T 31df5c2c40de4d88b6ab1567b84ce4ef P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 1.
I20251023 20:15:38.984946 23786 tablet_bootstrap.cc:492] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.985139 23786 ts_tablet_manager.cc:1403] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.001s
I20251023 20:15:38.985805 23786 raft_consensus.cc:359] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.985991 23786 raft_consensus.cc:385] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:38.986053 23786 raft_consensus.cc:740] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:38.986279 23786 consensus_queue.cc:260] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:38.986861 23786 ts_tablet_manager.cc:1434] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:38.987142 23786 tablet_bootstrap.cc:492] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:38.988958 23786 tablet_bootstrap.cc:654] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.997890 23786 tablet_bootstrap.cc:492] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:38.998494 23786 ts_tablet_manager.cc:1403] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20251023 20:15:38.999312 25784 consensus_queue.cc:1048] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.000792 24780 ts_tablet_manager.cc:1929] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.000924 24780 log.cc:1199] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/6480cdc1afda4671a346044f232b083e
I20251023 20:15:38.999323 23786 raft_consensus.cc:359] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.001552 23786 raft_consensus.cc:385] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.001744 23786 raft_consensus.cc:740] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.002120 23786 consensus_queue.cc:260] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.003062 23786 ts_tablet_manager.cc:1434] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251023 20:15:39.003773 23786 tablet_bootstrap.cc:492] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.005613 23786 tablet_bootstrap.cc:654] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.006148 24780 ts_tablet_manager.cc:1950] T 6480cdc1afda4671a346044f232b083e P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.007302 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 218) took 7972 ms. Trace:
I20251023 20:15:39.008049 25739 raft_consensus.cc:493] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.008224 25739 raft_consensus.cc:515] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.008455 24780 rpcz_store.cc:276] 1023 20:15:31.034441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034834 (+ 393us) service_pool.cc:225] Handling call
1023 20:15:38.962002 (+7927168us) ts_tablet_manager.cc:1151] Deleting tablet 6480cdc1afda4671a346044f232b083e
1023 20:15:38.962009 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.997773 (+ 35764us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.000686 (+ 2913us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.007278 (+ 6592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7925475}
I20251023 20:15:39.009228 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 214) took 7975 ms. Trace:
I20251023 20:15:39.009684 24780 rpcz_store.cc:276] 1023 20:15:31.034123 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034754 (+ 631us) service_pool.cc:225] Handling call
1023 20:15:39.009122 (+7974368us) ts_tablet_manager.cc:1151] Deleting tablet f103bece2fa240c19c23ac51046c13ed
1023 20:15:39.009134 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.009211 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7972449}
I20251023 20:15:39.009879 25739 leader_election.cc:290] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.010231 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 217) took 7975 ms. Trace:
I20251023 20:15:39.010246 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.010502 23667 raft_consensus.cc:2468] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.010586 24780 rpcz_store.cc:276] 1023 20:15:31.034365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034496 (+ 131us) service_pool.cc:225] Handling call
1023 20:15:39.010149 (+7975653us) ts_tablet_manager.cc:1151] Deleting tablet 8da82e6d24b14a129b91bb5298139d07
1023 20:15:39.010158 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.010215 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7971403}
I20251023 20:15:39.010910 23470 leader_election.cc:304] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.011121 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 224) took 7972 ms. Trace:
I20251023 20:15:39.011307 25167 raft_consensus.cc:2804] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.011422 25167 raft_consensus.cc:493] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.011445 24780 rpcz_store.cc:276] 1023 20:15:31.038362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.038475 (+ 113us) service_pool.cc:225] Handling call
1023 20:15:39.011047 (+7972572us) ts_tablet_manager.cc:1151] Deleting tablet d9f422575c7c4251879130501ab721fd
1023 20:15:39.011055 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.011107 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7972169}
I20251023 20:15:39.011529 25167 raft_consensus.cc:3060] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.011909 24780 tablet_replica.cc:333] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.012305 24780 raft_consensus.cc:2243] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.012722 24780 raft_consensus.cc:2272] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.013542 24780 ts_tablet_manager.cc:1916] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.014225 25167 raft_consensus.cc:515] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.016044 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.016271 23667 raft_consensus.cc:3060] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.018922 23667 raft_consensus.cc:2468] T 3e7909dba52748bdb78e08fa4b8a2d47 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.019500 23470 leader_election.cc:304] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.019909 25739 raft_consensus.cc:2804] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.020074 25739 raft_consensus.cc:697] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.020310 25492 ts_tablet_manager.cc:1929] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251023 20:15:39.020589 25911 log.cc:927] Time spent T f9bd2b109bfa4804a372965e60ba6e85 P d220aba7c9c0400398ffde917376ee48: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20251023 20:15:39.024214 23949 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6545140d94b7485aa81c51ad233355cd -> Peer 690ffde75cd745b6a87925967d54471a (127.22.127.188:41417): Couldn't send request to peer 690ffde75cd745b6a87925967d54471a. Status: Network error: Client connection negotiation failed: client connection to 127.22.127.188:41417: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.025216 24780 ts_tablet_manager.cc:1929] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:39.025923 23787 tablet_bootstrap.cc:492] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.026044 23790 tablet_bootstrap.cc:492] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:39.026051 23787 ts_tablet_manager.cc:1403] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.172s user 0.024s sys 0.012s
I20251023 20:15:39.026245 23790 ts_tablet_manager.cc:1403] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.140s user 0.007s sys 0.015s
I20251023 20:15:39.026819 23787 raft_consensus.cc:359] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.026973 23787 raft_consensus.cc:385] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.027024 23787 raft_consensus.cc:740] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.027199 23787 consensus_queue.cc:260] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.027688 23787 ts_tablet_manager.cc:1434] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:39.029330 23787 tablet_bootstrap.cc:492] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.031001 23787 tablet_bootstrap.cc:654] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:38.956835 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7366a659fef84d17925d61af9a4d456c [id=910ad9776933488eb34806b4aaff92d8]
I20251023 20:15:39.031975 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_12e6cb242b54467f9041afc60fa47eb3 [id=91a585d91b924838996b49fe349985f0]
I20251023 20:15:39.032806 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_64ad686282f94cb7ac869f59933669f1 [id=91b7c90942ea4aa8bb8e4b32e6d7c5db]
I20251023 20:15:39.035856 24297 consensus_queue.cc:1048] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.038194 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_61af7ade632c46a3becb1573b6313f0c [id=92ce6ecbc3fd4b48af6db9e587adcac5]
I20251023 20:15:39.038961 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_10bf526d2893408887984427efbaa1f2 [id=933b756e69bf4d6d96590acd69f45267]
I20251023 20:15:39.040766 25592 consensus_queue.cc:1048] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.043987 25282 ts_tablet_manager.cc:1929] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:39.044229 25282 log.cc:1199] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/5acdc66df9c748a1a6300962f62fcf36
I20251023 20:15:39.045653 23787 tablet_bootstrap.cc:492] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.045790 23787 ts_tablet_manager.cc:1403] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.005s
I20251023 20:15:39.045921 23790 raft_consensus.cc:359] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.046154 23790 raft_consensus.cc:385] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.046250 23790 raft_consensus.cc:740] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.046484 23790 consensus_queue.cc:260] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.046566 23787 raft_consensus.cc:359] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.046746 23787 raft_consensus.cc:385] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.046800 23787 raft_consensus.cc:740] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.047008 23787 consensus_queue.cc:260] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.047487 23787 ts_tablet_manager.cc:1434] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:39.047784 23787 tablet_bootstrap.cc:492] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.048110 23790 ts_tablet_manager.cc:1434] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251023 20:15:39.049206 23790 tablet_bootstrap.cc:492] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.049499 23787 tablet_bootstrap.cc:654] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.050997 23790 tablet_bootstrap.cc:654] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.052449 24780 log.cc:1199] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/6a54470d088748b0be22ab9db5a25f87
I20251023 20:15:39.052795 24780 ts_tablet_manager.cc:1950] T 6a54470d088748b0be22ab9db5a25f87 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.053581 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 222) took 8015 ms. Trace:
I20251023 20:15:39.053710 24780 rpcz_store.cc:276] 1023 20:15:31.038207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.038401 (+ 194us) service_pool.cc:225] Handling call
1023 20:15:39.011864 (+7973463us) ts_tablet_manager.cc:1151] Deleting tablet 6a54470d088748b0be22ab9db5a25f87
1023 20:15:39.011872 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.019960 (+ 8088us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.024345 (+ 4385us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.053562 (+ 29217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7971374}
I20251023 20:15:39.054139 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 219) took 8019 ms. Trace:
I20251023 20:15:39.054251 24780 rpcz_store.cc:276] 1023 20:15:31.034523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034577 (+ 54us) service_pool.cc:225] Handling call
1023 20:15:39.054048 (+8019471us) ts_tablet_manager.cc:1151] Deleting tablet cec51c54ea534b1da4b1d170e79d9f1c
1023 20:15:39.054057 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.054126 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8013051}
I20251023 20:15:39.054569 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 208) took 8021 ms. Trace:
I20251023 20:15:39.054677 24780 rpcz_store.cc:276] 1023 20:15:31.033606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.035210 (+ 1604us) service_pool.cc:225] Handling call
1023 20:15:39.054499 (+8019289us) ts_tablet_manager.cc:1151] Deleting tablet 95574277dde44983bb0a43d2b5ad01d5
1023 20:15:39.054507 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.054554 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8013297}
I20251023 20:15:39.054960 24780 tablet_replica.cc:333] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.055174 24780 raft_consensus.cc:2243] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.055348 24780 raft_consensus.cc:2272] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.056272 24780 ts_tablet_manager.cc:1916] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.062731 24780 ts_tablet_manager.cc:1929] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.068562 24780 log.cc:1199] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a86a25659ec44d7c825783036342ed45
I20251023 20:15:39.069229 24780 ts_tablet_manager.cc:1950] T a86a25659ec44d7c825783036342ed45 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.069423 25282 ts_tablet_manager.cc:1950] T 5acdc66df9c748a1a6300962f62fcf36 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:39.070678 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 385) took 5512 ms. Trace:
I20251023 20:15:39.070801 25282 rpcz_store.cc:276] 1023 20:15:33.558434 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.558481 (+ 47us) service_pool.cc:225] Handling call
1023 20:15:38.798698 (+5240217us) ts_tablet_manager.cc:1151] Deleting tablet 5acdc66df9c748a1a6300962f62fcf36
1023 20:15:38.798710 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:38.930063 (+131353us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.043776 (+113713us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.070652 (+ 26876us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14479,"tablet-delete.queue_time_us":5240038}
I20251023 20:15:39.071220 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 386) took 5511 ms. Trace:
I20251023 20:15:39.071296 25282 rpcz_store.cc:276] 1023 20:15:33.559464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.559509 (+ 45us) service_pool.cc:225] Handling call
1023 20:15:39.071110 (+5511601us) ts_tablet_manager.cc:1151] Deleting tablet bb475e303d07491bbc19dc3f628bceb7
1023 20:15:39.071121 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.071204 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5511425}
I20251023 20:15:39.071548 25282 tablet_replica.cc:333] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:39.071755 25282 raft_consensus.cc:2243] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.071933 25282 raft_consensus.cc:2272] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.071753 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 209) took 8038 ms. Trace:
I20251023 20:15:39.072079 24780 rpcz_store.cc:276] 1023 20:15:31.033712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.035124 (+ 1412us) service_pool.cc:225] Handling call
1023 20:15:39.054924 (+8019800us) ts_tablet_manager.cc:1151] Deleting tablet a86a25659ec44d7c825783036342ed45
1023 20:15:39.054931 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.059776 (+ 4845us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.062639 (+ 2863us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.071727 (+ 9088us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8013625}
I20251023 20:15:39.072503 25282 ts_tablet_manager.cc:1916] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.072786 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 216) took 8038 ms. Trace:
I20251023 20:15:39.073105 24780 rpcz_store.cc:276] 1023 20:15:31.034295 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034779 (+ 484us) service_pool.cc:225] Handling call
1023 20:15:39.072693 (+8037914us) ts_tablet_manager.cc:1151] Deleting tablet 600f0110f8bd4b05983d5922d871c266
1023 20:15:39.072703 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.072771 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8031198}
I20251023 20:15:39.073565 24780 tablet_replica.cc:333] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
W20251023 20:15:39.073513 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 385) could send its response
W20251023 20:15:39.073685 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 386) could send its response
I20251023 20:15:39.074005 24780 raft_consensus.cc:2243] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.074602 24780 pending_rounds.cc:70] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:39.074991 24780 pending_rounds.cc:77] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082172714160128 op_type: NO_OP noop_request { }
I20251023 20:15:39.075256 24780 raft_consensus.cc:2889] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:39.075462 24780 raft_consensus.cc:2272] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.076159 24780 ts_tablet_manager.cc:1916] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.020356 25739 consensus_queue.cc:237] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.025219 25784 consensus_queue.cc:1048] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.042086 25466 raft_consensus.cc:493] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.082579 25466 raft_consensus.cc:515] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.082890 25492 log.cc:1199] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/26a680ad3aed487593bd1b63004dbbfc
I20251023 20:15:39.053544 25167 leader_election.cc:290] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.089521 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_4ad8e8f4f67e4639bcab0f4fd2c7b099 [id=94fc73e172904419b94d749aa2ef8dc9]
I20251023 20:15:39.090324 25492 ts_tablet_manager.cc:1950] T 26a680ad3aed487593bd1b63004dbbfc P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.080112 25926 raft_consensus.cc:493] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.096736 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.098172 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:39.098943 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_515e6170d0e34340a74759ab0a056f72 [id=95f5faaee2b947fea16c52c55657db4a]
W20251023 20:15:39.102272 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 218) could send its response
I20251023 20:15:39.104753 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 410) took 3788 ms. Trace:
I20251023 20:15:39.104920 25492 rpcz_store.cc:276] 1023 20:15:35.316096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.316152 (+ 56us) service_pool.cc:225] Handling call
1023 20:15:38.897319 (+3581167us) ts_tablet_manager.cc:1151] Deleting tablet 26a680ad3aed487593bd1b63004dbbfc
1023 20:15:38.897327 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.016949 (+119622us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.020224 (+ 3275us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.104732 (+ 84508us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6288,"tablet-delete.queue_time_us":3580953}
I20251023 20:15:39.105438 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 412) took 3789 ms. Trace:
W20251023 20:15:39.118404 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 214) could send its response
I20251023 20:15:39.118394 25926 raft_consensus.cc:515] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.103957 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4798f6eea5745ba8a0d4b8cb173cc5c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.118878 25466 leader_election.cc:290] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:39.120164 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 217) could send its response
I20251023 20:15:39.119149 25492 rpcz_store.cc:276] 1023 20:15:35.316302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.316404 (+ 102us) service_pool.cc:225] Handling call
1023 20:15:39.105346 (+3788942us) ts_tablet_manager.cc:1151] Deleting tablet 6a54470d088748b0be22ab9db5a25f87
1023 20:15:39.105356 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.105424 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3788812}
I20251023 20:15:39.120944 24780 ts_tablet_manager.cc:1929] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.121035 24780 log.cc:1199] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a8eaa2167f82475ca2f6a110aba288ec
I20251023 20:15:39.121199 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ea656dd8eee44e5f89ad208c70cddc39 [id=96188ab2961f4b14b71df728d91aa4e3]
I20251023 20:15:39.121261 25926 leader_election.cc:290] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.122730 23787 tablet_bootstrap.cc:492] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.123317 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 414) took 3806 ms. Trace:
I20251023 20:15:39.123806 23787 ts_tablet_manager.cc:1403] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.001s
I20251023 20:15:39.125356 25924 raft_consensus.cc:493] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251023 20:15:39.123529 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 224) could send its response
I20251023 20:15:39.125560 25924 raft_consensus.cc:515] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.125905 24780 ts_tablet_manager.cc:1950] T a8eaa2167f82475ca2f6a110aba288ec P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.127624 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 212) took 8093 ms. Trace:
I20251023 20:15:39.124671 25492 rpcz_store.cc:276] 1023 20:15:35.316490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.316560 (+ 70us) service_pool.cc:225] Handling call
1023 20:15:39.123205 (+3806645us) ts_tablet_manager.cc:1151] Deleting tablet a636a4cdeb404e9f806d8e166ea57938
1023 20:15:39.123218 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.123301 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3806511}
I20251023 20:15:39.128422 24780 rpcz_store.cc:276] 1023 20:15:31.033944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034674 (+ 730us) service_pool.cc:225] Handling call
1023 20:15:39.073518 (+8038844us) ts_tablet_manager.cc:1151] Deleting tablet a8eaa2167f82475ca2f6a110aba288ec
1023 20:15:39.073527 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.117955 (+ 44428us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.120841 (+ 2886us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.127605 (+ 6764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8031527}
I20251023 20:15:39.129033 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 221) took 8090 ms. Trace:
I20251023 20:15:39.129438 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 415) took 3812 ms. Trace:
I20251023 20:15:39.129436 24780 rpcz_store.cc:276] 1023 20:15:31.038106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.038424 (+ 318us) service_pool.cc:225] Handling call
1023 20:15:39.128933 (+8090509us) ts_tablet_manager.cc:1151] Deleting tablet af899c01b4af46d39d27f984cc3d5940
1023 20:15:39.128945 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.129017 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8085379}
I20251023 20:15:39.129951 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 223) took 8091 ms. Trace:
I20251023 20:15:39.130281 24780 rpcz_store.cc:276] 1023 20:15:31.038290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.038509 (+ 219us) service_pool.cc:225] Handling call
1023 20:15:39.129870 (+8091361us) ts_tablet_manager.cc:1151] Deleting tablet d82c92e863b2470699cc822b8cb8c14d
1023 20:15:39.129879 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.129937 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8085941}
I20251023 20:15:39.130733 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 215) took 8096 ms. Trace:
I20251023 20:15:39.131081 24780 rpcz_store.cc:276] 1023 20:15:31.034205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.034542 (+ 337us) service_pool.cc:225] Handling call
1023 20:15:39.130656 (+8096114us) ts_tablet_manager.cc:1151] Deleting tablet 523d72012c364457b9a2a299f2f789a1
1023 20:15:39.130665 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.130719 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8086271}
I20251023 20:15:39.131532 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 225) took 8090 ms. Trace:
I20251023 20:15:39.132006 24780 rpcz_store.cc:276] 1023 20:15:31.046516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.046792 (+ 276us) service_pool.cc:225] Handling call
1023 20:15:39.131455 (+8084663us) ts_tablet_manager.cc:1151] Deleting tablet c97e2e0f839a4ef389fe100b6a81301e
1023 20:15:39.131464 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.131517 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8084487}
I20251023 20:15:39.132906 24780 tablet_replica.cc:333] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.133272 24780 raft_consensus.cc:2243] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.133849 24780 pending_rounds.cc:70] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:39.134228 24780 pending_rounds.cc:77] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082174312988672 op_type: NO_OP noop_request { }
I20251023 20:15:39.134454 24780 raft_consensus.cc:2889] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:39.134644 24780 raft_consensus.cc:2272] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.134673 25854 raft_consensus.cc:493] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.134807 25854 raft_consensus.cc:515] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.135325 24780 ts_tablet_manager.cc:1916] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.135355 25854 leader_election.cc:290] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:39.136063 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc7793d1370d47ac8ee66d0246aba8fe" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.132366 23787 raft_consensus.cc:359] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:39.136592 23605 leader_election.cc:343] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.136915 23787 raft_consensus.cc:385] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.145558 24780 ts_tablet_manager.cc:1929] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.151053 24780 log.cc:1199] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/cbacaf2fea24405caaf84d59b625fca3
I20251023 20:15:39.151618 24780 ts_tablet_manager.cc:1950] T cbacaf2fea24405caaf84d59b625fca3 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.153918 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 226) took 8107 ms. Trace:
I20251023 20:15:39.154094 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_f232c9e65a8248d2b818d81486618f5f [id=9834a6ff7acf4269aebcae4712fc7149]
I20251023 20:15:39.154325 24780 rpcz_store.cc:276] 1023 20:15:31.046752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.047008 (+ 256us) service_pool.cc:225] Handling call
1023 20:15:39.132853 (+8085845us) ts_tablet_manager.cc:1151] Deleting tablet cbacaf2fea24405caaf84d59b625fca3
1023 20:15:39.132863 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.142483 (+ 9620us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.145456 (+ 2973us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.153895 (+ 8439us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8085427}
I20251023 20:15:39.154973 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b]
I20251023 20:15:39.154964 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 227) took 8107 ms. Trace:
I20251023 20:15:39.155355 24780 rpcz_store.cc:276] 1023 20:15:31.047277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.047315 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:39.154864 (+8107549us) ts_tablet_manager.cc:1151] Deleting tablet 1a64a7c2e1f242e1b54243489622bae5
1023 20:15:39.154875 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.154945 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8107371}
I20251023 20:15:39.155704 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6]
I20251023 20:15:39.155988 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 228) took 8108 ms. Trace:
I20251023 20:15:39.156337 24780 rpcz_store.cc:276] 1023 20:15:31.047599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.047720 (+ 121us) service_pool.cc:225] Handling call
1023 20:15:39.155899 (+8108179us) ts_tablet_manager.cc:1151] Deleting tablet 83dae34129794f49b43038d3b1636ea0
1023 20:15:39.155910 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.155970 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8108023}
I20251023 20:15:39.156823 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_322349f14bf94a7cbd9cb6cb92793905 [id=9ae6a893fdef4f328cbfd0ee2d57f5a1]
I20251023 20:15:39.157162 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 229) took 8109 ms. Trace:
I20251023 20:15:39.130879 23786 tablet_bootstrap.cc:492] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.157382 24780 rpcz_store.cc:276] 1023 20:15:31.047687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.047916 (+ 229us) service_pool.cc:225] Handling call
1023 20:15:39.156822 (+8108906us) ts_tablet_manager.cc:1151] Deleting tablet 1e5720c96730425dafb550094cf936cd
1023 20:15:39.156831 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.157142 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8108726}
I20251023 20:15:39.157619 23786 ts_tablet_manager.cc:1403] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.154s user 0.022s sys 0.031s
I20251023 20:15:39.157649 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 230) took 8109 ms. Trace:
I20251023 20:15:39.157708 24780 rpcz_store.cc:276] 1023 20:15:31.048210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.048257 (+ 47us) service_pool.cc:225] Handling call
1023 20:15:39.157577 (+8109320us) ts_tablet_manager.cc:1151] Deleting tablet a9c5b69df4c04365b1ac1457fe8300cc
1023 20:15:39.157585 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.157635 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8109170}
I20251023 20:15:39.157956 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 232) took 8109 ms. Trace:
I20251023 20:15:39.158016 24780 rpcz_store.cc:276] 1023 20:15:31.048927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.049043 (+ 116us) service_pool.cc:225] Handling call
1023 20:15:39.157892 (+8108849us) ts_tablet_manager.cc:1151] Deleting tablet 5e73ae95c73945f081c298f49d47c904
1023 20:15:39.157899 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.157942 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8108455}
I20251023 20:15:39.158257 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 233) took 8109 ms. Trace:
I20251023 20:15:39.158315 24780 rpcz_store.cc:276] 1023 20:15:31.049009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.049149 (+ 140us) service_pool.cc:225] Handling call
1023 20:15:39.158187 (+8109038us) ts_tablet_manager.cc:1151] Deleting tablet a636a4cdeb404e9f806d8e166ea57938
1023 20:15:39.158195 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.158244 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8108608}
I20251023 20:15:39.158437 23786 raft_consensus.cc:359] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.158561 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 234) took 8109 ms. Trace:
I20251023 20:15:39.158604 23786 raft_consensus.cc:385] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.158658 23786 raft_consensus.cc:740] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.158619 24780 rpcz_store.cc:276] 1023 20:15:31.049269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.049606 (+ 337us) service_pool.cc:225] Handling call
1023 20:15:39.158494 (+8108888us) ts_tablet_manager.cc:1151] Deleting tablet f908f086946949829cac15418a584730
1023 20:15:39.158502 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.158548 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8108633}
I20251023 20:15:39.158835 24780 tablet_replica.cc:333] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.135502 25924 leader_election.cc:290] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:39.158954 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 222) could send its response
W20251023 20:15:39.159101 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 219) could send its response
W20251023 20:15:39.159193 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 208) could send its response
I20251023 20:15:39.159499 23786 consensus_queue.cc:260] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.160013 23786 ts_tablet_manager.cc:1434] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:39.160281 23786 tablet_bootstrap.cc:492] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.161917 23786 tablet_bootstrap.cc:654] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.159003 24780 raft_consensus.cc:2243] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.162258 24780 raft_consensus.cc:2272] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.162411 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc7793d1370d47ac8ee66d0246aba8fe" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.162786 24780 ts_tablet_manager.cc:1916] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:39.137277 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 410) could send its response
W20251023 20:15:39.163496 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 412) could send its response
I20251023 20:15:39.163618 23790 tablet_bootstrap.cc:492] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
W20251023 20:15:39.163686 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 414) could send its response
I20251023 20:15:39.163817 23790 ts_tablet_manager.cc:1403] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.026s
I20251023 20:15:39.164479 23790 raft_consensus.cc:359] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.164683 23790 raft_consensus.cc:385] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.164767 23790 raft_consensus.cc:740] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.165009 23790 consensus_queue.cc:260] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:39.170177 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 209) could send its response
W20251023 20:15:39.170332 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 216) could send its response
I20251023 20:15:39.170902 25924 consensus_queue.cc:1048] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.172060 25924 consensus_queue.cc:1048] T b446550686024c3ca80f106112a2110f P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.172479 25924 consensus_queue.cc:1048] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.177484 23790 ts_tablet_manager.cc:1434] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20251023 20:15:39.178678 23790 tablet_bootstrap.cc:492] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.186340 25739 consensus_queue.cc:1048] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.187918 24780 ts_tablet_manager.cc:1929] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.188035 24780 log.cc:1199] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/e194d03a6aa946fbb2c79d91b196440b
W20251023 20:15:39.188258 23471 leader_election.cc:343] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.188367 24780 ts_tablet_manager.cc:1950] T e194d03a6aa946fbb2c79d91b196440b P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
W20251023 20:15:39.188800 23471 leader_election.cc:343] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.189153 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 235) took 8139 ms. Trace:
I20251023 20:15:39.189453 23790 tablet_bootstrap.cc:654] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.189396 25925 consensus_queue.cc:1048] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.191521 25594 raft_consensus.cc:493] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.191709 25594 raft_consensus.cc:515] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.192258 25594 leader_election.cc:290] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.192884 25921 consensus_queue.cc:1048] T cd548fd9de284377ab557bd98f621068 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:39.193496 23471 leader_election.cc:343] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:39.193913 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 212) could send its response
I20251023 20:15:39.192891 25924 consensus_queue.cc:1048] T 59598ea40b9e4e5da0ec36f35f1094d6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20251023 20:15:39.194175 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 221) could send its response
W20251023 20:15:39.194355 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 223) could send its response
W20251023 20:15:39.194499 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 215) could send its response
W20251023 20:15:39.194692 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 225) could send its response
I20251023 20:15:39.157495 25492 rpcz_store.cc:276] 1023 20:15:35.316566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.316714 (+ 148us) service_pool.cc:225] Handling call
1023 20:15:39.129329 (+3812615us) ts_tablet_manager.cc:1151] Deleting tablet af899c01b4af46d39d27f984cc3d5940
1023 20:15:39.129339 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.129420 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3812191}
W20251023 20:15:39.197263 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 226) could send its response
W20251023 20:15:39.197516 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 227) could send its response
W20251023 20:15:39.197675 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 228) could send its response
W20251023 20:15:39.197817 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 415) could send its response
W20251023 20:15:39.197850 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 229) could send its response
W20251023 20:15:39.197999 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 230) could send its response
W20251023 20:15:39.198141 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 232) could send its response
I20251023 20:15:39.198225 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 416) took 3881 ms. Trace:
W20251023 20:15:39.198333 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 233) could send its response
W20251023 20:15:39.198537 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 234) could send its response
I20251023 20:15:39.198487 25492 rpcz_store.cc:276] 1023 20:15:35.316681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.316739 (+ 58us) service_pool.cc:225] Handling call
1023 20:15:39.197369 (+3880630us) ts_tablet_manager.cc:1151] Deleting tablet f103bece2fa240c19c23ac51046c13ed
1023 20:15:39.197380 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.198211 (+ 831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3880424}
I20251023 20:15:39.199111 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 413) took 3882 ms. Trace:
I20251023 20:15:39.199517 25492 rpcz_store.cc:276] 1023 20:15:35.316411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.316855 (+ 444us) service_pool.cc:225] Handling call
1023 20:15:39.199033 (+3882178us) ts_tablet_manager.cc:1151] Deleting tablet 600f0110f8bd4b05983d5922d871c266
1023 20:15:39.199042 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.199097 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3882075}
I20251023 20:15:39.199990 25492 tablet_replica.cc:333] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.200369 25492 raft_consensus.cc:2243] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.202046 25492 raft_consensus.cc:2272] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.145542 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4798f6eea5745ba8a0d4b8cb173cc5c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.202360 23668 raft_consensus.cc:2468] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.156869 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9749c6f3fe5545ab93bf49aae97e9818" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.153232 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e5dc42c01a4d5e9661f399f793f68f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.202661 23669 raft_consensus.cc:2468] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.202731 23663 raft_consensus.cc:2468] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.202950 25492 ts_tablet_manager.cc:1916] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.138012 23787 raft_consensus.cc:740] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.194443 24780 rpcz_store.cc:276] 1023 20:15:31.049990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.050030 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:39.158784 (+8108754us) ts_tablet_manager.cc:1151] Deleting tablet e194d03a6aa946fbb2c79d91b196440b
1023 20:15:39.158792 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.184877 (+ 26085us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.187822 (+ 2945us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.189133 (+ 1311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8108593}
I20251023 20:15:39.203626 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 236) took 8153 ms. Trace:
W20251023 20:15:39.199215 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 416) could send its response
I20251023 20:15:39.203738 24780 rpcz_store.cc:276] 1023 20:15:31.050293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.050328 (+ 35us) service_pool.cc:225] Handling call
1023 20:15:39.203514 (+8153186us) ts_tablet_manager.cc:1151] Deleting tablet 46f700ea3db04a4bad539c943e850a44
1023 20:15:39.203526 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.203610 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8153032}
W20251023 20:15:39.203897 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 413) could send its response
I20251023 20:15:39.203706 23787 consensus_queue.cc:260] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.204089 24780 tablet_replica.cc:333] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.204608 24780 raft_consensus.cc:2243] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.204909 24780 raft_consensus.cc:2272] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.205948 24780 ts_tablet_manager.cc:1916] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.208380 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7]
I20251023 20:15:39.208844 25854 raft_consensus.cc:493] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.208961 25854 raft_consensus.cc:515] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.209154 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_fbff76a175884e39a9efd67edd4db37e [id=9ec5b95ed1e340a784ea0a4cbf6e2e7b]
I20251023 20:15:39.209503 25854 leader_election.cc:290] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:39.210397 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6313d3307aff4a6880176a06627e86fc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:39.211068 23605 leader_election.cc:343] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:39.213375 23470 consensus_peers.cc:597] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9bd2b109bfa4804a372965e60ba6e85. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.214305 25921 consensus_queue.cc:1048] T 4bce63866d524067b87145b07347ffcf P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.215373 25854 raft_consensus.cc:493] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.215495 25854 raft_consensus.cc:515] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.216722 25921 consensus_queue.cc:1048] T 19da816a483a4d5b9dc768ef697d54ec P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.217478 23786 tablet_bootstrap.cc:492] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.217669 23786 ts_tablet_manager.cc:1403] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.005s
W20251023 20:15:39.218468 23470 consensus_peers.cc:597] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a6a2d2b25824dbdafaf2a9cabb0f061. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.218874 23786 raft_consensus.cc:359] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.219069 23786 raft_consensus.cc:385] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.219157 23786 raft_consensus.cc:740] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.219412 23786 consensus_queue.cc:260] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.220250 23787 ts_tablet_manager.cc:1434] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.096s user 0.001s sys 0.002s
I20251023 20:15:39.222507 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cdde6b251b14df4bf44142ce263b84a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.222769 23527 raft_consensus.cc:2393] T 5cdde6b251b14df4bf44142ce263b84a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate 9bc6522db11e4960a9aad84435371a32 in this term.
I20251023 20:15:39.223502 25594 raft_consensus.cc:493] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.223655 25594 raft_consensus.cc:515] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.223687 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5cdde6b251b14df4bf44142ce263b84a" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.224249 25594 leader_election.cc:290] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.225975 23786 ts_tablet_manager.cc:1434] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:39.226238 23786 tablet_bootstrap.cc:492] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.227617 23786 tablet_bootstrap.cc:654] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.227900 25276 raft_consensus.cc:493] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.228051 25276 raft_consensus.cc:515] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.229141 25492 ts_tablet_manager.cc:1929] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.230965 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e1d99e08ee354f119f078584381958e6 [id=a34d678a3e1f41d8b65903bc2126e4e8]
I20251023 20:15:39.231977 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75]
I20251023 20:15:39.233376 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_03569553d00b42c890bd05b75ff28289 [id=a5b078d079dc454ab853c988889beb21]
I20251023 20:15:39.234505 25776 raft_consensus.cc:493] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.234649 25776 raft_consensus.cc:515] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:39.234402 23333 consensus_peers.cc:597] T 1822fdf1f5f44dd4a742aa3c2f8443cf P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1822fdf1f5f44dd4a742aa3c2f8443cf. This is attempt 1: this message will repeat every 5th retry.
W20251023 20:15:39.236665 23333 consensus_peers.cc:597] T 01fa5f9c700b458ea147373d78551573 P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01fa5f9c700b458ea147373d78551573. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.237110 25592 consensus_queue.cc:1048] T a2c022d78cac4959984d608d64df851b P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.238492 25489 consensus_queue.cc:1048] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:39.239965 23333 consensus_peers.cc:597] T 377de4ea0bd6489284fac6b0fab52c8c P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 377de4ea0bd6489284fac6b0fab52c8c. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.240981 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e5dc42c01a4d5e9661f399f793f68f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.244102 25492 log.cc:1199] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/6480cdc1afda4671a346044f232b083e
I20251023 20:15:39.244454 25492 ts_tablet_manager.cc:1950] T 6480cdc1afda4671a346044f232b083e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.247805 25854 leader_election.cc:290] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:39.248487 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38ce80a338d2494aac84fe6344d47e90" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:39.249044 23605 leader_election.cc:343] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.252593 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 411) took 3928 ms. Trace:
I20251023 20:15:39.252820 25492 rpcz_store.cc:276] 1023 20:15:35.316200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.316970 (+ 770us) service_pool.cc:225] Handling call
1023 20:15:39.199945 (+3882975us) ts_tablet_manager.cc:1151] Deleting tablet 6480cdc1afda4671a346044f232b083e
1023 20:15:39.199953 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.207790 (+ 7837us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.229013 (+ 21223us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.245157 (+ 16144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3882814}
W20251023 20:15:39.253425 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 411) could send its response
I20251023 20:15:39.253782 25492 tablet_replica.cc:333] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.254109 25492 raft_consensus.cc:2243] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.254467 25492 raft_consensus.cc:2272] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.255144 25492 ts_tablet_manager.cc:1916] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.256444 25276 leader_election.cc:290] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:39.256976 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "878508d6bf0d4a59b6f64155361b4418" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.257341 25776 leader_election.cc:290] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
W20251023 20:15:39.257562 23605 leader_election.cc:343] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.257941 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:39.258390 23605 leader_election.cc:343] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.270190 23790 tablet_bootstrap.cc:492] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:39.270346 23790 ts_tablet_manager.cc:1403] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.092s user 0.020s sys 0.026s
I20251023 20:15:39.271121 23790 raft_consensus.cc:359] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.271301 23790 raft_consensus.cc:385] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.271347 23790 raft_consensus.cc:740] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.271559 23790 consensus_queue.cc:260] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.272055 23790 ts_tablet_manager.cc:1434] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251023 20:15:39.272308 23790 tablet_bootstrap.cc:492] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.273957 23790 tablet_bootstrap.cc:654] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.277032 23787 tablet_bootstrap.cc:492] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.279361 23787 tablet_bootstrap.cc:654] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.280468 23790 tablet_bootstrap.cc:492] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:39.280607 23790 ts_tablet_manager.cc:1403] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20251023 20:15:39.281283 23790 raft_consensus.cc:359] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.281443 23790 raft_consensus.cc:385] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.281495 23790 raft_consensus.cc:740] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.282142 25282 ts_tablet_manager.cc:1929] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.282382 25282 log.cc:1199] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/6480cdc1afda4671a346044f232b083e
I20251023 20:15:39.282205 23790 consensus_queue.cc:260] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.282908 25282 ts_tablet_manager.cc:1950] T 6480cdc1afda4671a346044f232b083e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:39.283085 23790 ts_tablet_manager.cc:1434] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:39.283604 23790 tablet_bootstrap.cc:492] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.283628 25492 ts_tablet_manager.cc:1929] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.283728 25492 log.cc:1199] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/40c659c273c449939d0dd708d39b393f
I20251023 20:15:39.284032 25492 ts_tablet_manager.cc:1950] T 40c659c273c449939d0dd708d39b393f P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.284865 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9749c6f3fe5545ab93bf49aae97e9818" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.288177 23396 raft_consensus.cc:3060] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.288650 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 417) took 3938 ms. Trace:
I20251023 20:15:39.288740 25492 rpcz_store.cc:276] 1023 20:15:35.350031 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.350093 (+ 62us) service_pool.cc:225] Handling call
1023 20:15:39.253731 (+3903638us) ts_tablet_manager.cc:1151] Deleting tablet 40c659c273c449939d0dd708d39b393f
1023 20:15:39.253741 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.262998 (+ 9257us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.283517 (+ 20519us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.288630 (+ 5113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3903373}
W20251023 20:15:39.288964 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 417) could send its response
I20251023 20:15:39.290253 25492 tablet_replica.cc:333] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.290454 25492 raft_consensus.cc:2243] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.235168 23470 leader_election.cc:304] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters: d220aba7c9c0400398ffde917376ee48
I20251023 20:15:39.290896 25927 raft_consensus.cc:2804] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.290963 25927 raft_consensus.cc:493] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.291025 25927 raft_consensus.cc:3060] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.291520 25776 consensus_queue.cc:1048] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.293382 25927 raft_consensus.cc:515] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.293921 25927 leader_election.cc:290] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.296617 24780 ts_tablet_manager.cc:1929] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.296852 23786 tablet_bootstrap.cc:492] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.297031 23786 ts_tablet_manager.cc:1403] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.006s
I20251023 20:15:39.297729 23786 raft_consensus.cc:359] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.297937 23786 raft_consensus.cc:385] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.298028 23786 raft_consensus.cc:740] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.298305 23786 consensus_queue.cc:260] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.298439 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 387) took 5728 ms. Trace:
I20251023 20:15:39.300036 23786 ts_tablet_manager.cc:1434] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251023 20:15:39.300930 23786 tablet_bootstrap.cc:492] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.299247 25282 rpcz_store.cc:276] 1023 20:15:33.570429 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.570487 (+ 58us) service_pool.cc:225] Handling call
1023 20:15:39.071505 (+5501018us) ts_tablet_manager.cc:1151] Deleting tablet 6480cdc1afda4671a346044f232b083e
1023 20:15:39.071513 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.126634 (+ 55121us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.282036 (+155402us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.298413 (+ 16377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5500813}
W20251023 20:15:39.301484 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 387) could send its response
I20251023 20:15:39.303020 23786 tablet_bootstrap.cc:654] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.303172 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 389) took 5730 ms. Trace:
I20251023 20:15:39.304123 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6]
I20251023 20:15:39.303767 25282 rpcz_store.cc:276] 1023 20:15:33.572420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.572462 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:39.301835 (+5729373us) ts_tablet_manager.cc:1151] Deleting tablet cbacaf2fea24405caaf84d59b625fca3
1023 20:15:39.302083 (+ 248us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.302919 (+ 836us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5728957}
I20251023 20:15:39.306484 25282 tablet_replica.cc:333] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
W20251023 20:15:39.308018 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 389) could send its response
I20251023 20:15:39.311642 25282 raft_consensus.cc:2243] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.311960 25282 raft_consensus.cc:2272] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.315212 25282 ts_tablet_manager.cc:1916] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.290678 25492 pending_rounds.cc:70] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Trying to abort 1 pending ops.
I20251023 20:15:39.317958 25492 pending_rounds.cc:77] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082204313169920 op_type: NO_OP noop_request { }
I20251023 20:15:39.318089 25492 raft_consensus.cc:2889] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:39.318194 25492 raft_consensus.cc:2272] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.318750 25492 ts_tablet_manager.cc:1916] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:39.290966 23470 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 55826.9us
I20251023 20:15:39.319396 23470 leader_election.cc:304] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.321440 25927 raft_consensus.cc:2804] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.321517 25927 raft_consensus.cc:493] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.321588 25927 raft_consensus.cc:3060] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.324069 25927 raft_consensus.cc:515] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.324874 25927 leader_election.cc:290] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.325320 23470 leader_election.cc:304] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.325610 25927 raft_consensus.cc:2804] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.325714 25927 raft_consensus.cc:493] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.325824 25927 raft_consensus.cc:3060] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.328142 25927 raft_consensus.cc:515] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.328888 25927 leader_election.cc:290] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:39.329382 23604 leader_election.cc:343] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.329502 23604 leader_election.cc:304] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.329859 25776 raft_consensus.cc:2749] T bc7793d1370d47ac8ee66d0246aba8fe P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.339742 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_1d7f65ff02bb4fccb1a45097c5036126 [id=a8c797d0049e4d1c9123db93f7ea12be]
I20251023 20:15:39.339947 23786 tablet_bootstrap.cc:492] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.340142 23786 ts_tablet_manager.cc:1403] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.003s
I20251023 20:15:39.341540 23786 raft_consensus.cc:359] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.341923 23786 raft_consensus.cc:385] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.342121 23786 raft_consensus.cc:740] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.342573 23786 consensus_queue.cc:260] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.345342 23786 ts_tablet_manager.cc:1434] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:39.352183 23787 tablet_bootstrap.cc:492] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.352370 23787 ts_tablet_manager.cc:1403] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.002s
I20251023 20:15:39.353206 23787 raft_consensus.cc:359] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.353426 23787 raft_consensus.cc:385] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.354190 23787 raft_consensus.cc:740] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.354789 23787 consensus_queue.cc:260] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.355929 23787 ts_tablet_manager.cc:1434] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:39.357124 23786 tablet_bootstrap.cc:492] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.359414 23787 tablet_bootstrap.cc:492] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.313477 23392 raft_consensus.cc:3060] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.362612 24780 log.cc:1199] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/ca11ddd5d97544cf916b87fd3dff219d
I20251023 20:15:39.330758 25594 raft_consensus.cc:493] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.364360 25594 raft_consensus.cc:515] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.364899 25594 leader_election.cc:290] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.346549 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4798f6eea5745ba8a0d4b8cb173cc5c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:39.347090 23387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9749c6f3fe5545ab93bf49aae97e9818" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:39.347265 23388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e5dc42c01a4d5e9661f399f793f68f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:39.367558 25492 ts_tablet_manager.cc:1929] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.369226 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4798f6eea5745ba8a0d4b8cb173cc5c" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.369385 23667 raft_consensus.cc:3060] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.371282 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7f5474847aa74113aca18fe96357be62 [id=ac08261334284759a3997df44867cc31]
I20251023 20:15:39.372049 23667 raft_consensus.cc:2468] T b4798f6eea5745ba8a0d4b8cb173cc5c P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.371949 24780 ts_tablet_manager.cc:1950] T ca11ddd5d97544cf916b87fd3dff219d P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.375641 25594 consensus_queue.cc:1048] T 98a6a3425a49403e99c8ade114166e46 P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.378649 23787 tablet_bootstrap.cc:654] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.379354 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8]
I20251023 20:15:39.380730 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13]
I20251023 20:15:39.381366 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9749c6f3fe5545ab93bf49aae97e9818" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.381572 23667 raft_consensus.cc:3060] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.382328 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_20937abab3a8462cb783c7a4f34a9600 [id=afc43a90036d4b9cb050f975473f2e56]
I20251023 20:15:39.383977 23667 raft_consensus.cc:2468] T 9749c6f3fe5545ab93bf49aae97e9818 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.389046 23667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e5dc42c01a4d5e9661f399f793f68f" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.389278 23667 raft_consensus.cc:3060] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.389456 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_c2d9a9353b6944648cb319a6ea0cf2f6 [id=afe5584c068540c1915787d9b87cab33]
I20251023 20:15:39.390283 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d]
I20251023 20:15:39.392129 23667 raft_consensus.cc:2468] T a9e5dc42c01a4d5e9661f399f793f68f P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.394500 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d]
I20251023 20:15:39.396497 23790 tablet_bootstrap.cc:654] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.398768 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_000ee28be94d48c394515c756bfe16bf [id=b1ab817cbd9e47ea93ac15403fbb35a9]
I20251023 20:15:39.400257 25282 ts_tablet_manager.cc:1929] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:39.400820 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_316f315bbf2a4332a958a5f72ec606c1 [id=b4bbb2644eaf438f82f912407adc031f]
I20251023 20:15:39.400699 25282 log.cc:1199] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/1e5720c96730425dafb550094cf936cd
I20251023 20:15:39.401273 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e2c91fae0bf148f198063a62b0426f63 [id=b4c7610c982345c4a6f90e2025b41b2d]
I20251023 20:15:39.401690 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973]
I20251023 20:15:39.402094 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_9c045162e1824edbbcf0718c49e3b185 [id=b6a25cca1d8e48af8e0791a1e879d6a8]
I20251023 20:15:39.402493 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae]
I20251023 20:15:39.402908 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_2860d1804bda4f67b048ee52ba46f15e [id=b9cb00bd8954475b909036fd9f123b1b]
I20251023 20:15:39.402890 25282 ts_tablet_manager.cc:1950] T 1e5720c96730425dafb550094cf936cd P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:39.403301 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_15836e4048de43f79d9a1d06e0072c3c [id=b9d64b8c30014c5abbb2312d1b8017fe]
I20251023 20:15:39.403908 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_3abcf10252074989b84e5566c762a99d [id=ba74b6d39572458e9f01923f709c2d38]
I20251023 20:15:39.404315 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_b589e93e3314462d8ef529985314e501 [id=bdeb476ba31d425893262ab98d5b579f]
I20251023 20:15:39.408782 23787 tablet_bootstrap.cc:492] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.410147 23787 ts_tablet_manager.cc:1403] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20251023 20:15:39.410852 23787 raft_consensus.cc:359] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.423141 23333 leader_election.cc:304] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.423439 25726 raft_consensus.cc:2749] T 5cdde6b251b14df4bf44142ce263b84a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:39.425045 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 235) could send its response
W20251023 20:15:39.426755 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 236) could send its response
I20251023 20:15:39.425595 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 237) took 8375 ms. Trace:
I20251023 20:15:39.428396 24780 rpcz_store.cc:276] 1023 20:15:31.050593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.050628 (+ 35us) service_pool.cc:225] Handling call
1023 20:15:39.204042 (+8153414us) ts_tablet_manager.cc:1151] Deleting tablet ca11ddd5d97544cf916b87fd3dff219d
1023 20:15:39.204051 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.229977 (+ 25926us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.296497 (+ 66520us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.425576 (+129079us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":410,"tablet-delete.queue_time_us":8153255}
I20251023 20:15:39.428839 24780 tablet_replica.cc:333] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.429339 24780 raft_consensus.cc:2243] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.429962 24780 raft_consensus.cc:2272] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.382321 23786 tablet_bootstrap.cc:654] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.430492 25492 log.cc:1199] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/857da3642f0846ce92aaed0491e89644
I20251023 20:15:39.387312 25927 raft_consensus.cc:493] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.393350 25739 raft_consensus.cc:493] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.431159 25739 raft_consensus.cc:515] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.393783 23527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6313d3307aff4a6880176a06627e86fc" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.395453 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38ce80a338d2494aac84fe6344d47e90" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.396054 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "878508d6bf0d4a59b6f64155361b4418" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.396381 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.427896 25924 consensus_queue.cc:1048] T 2202632a57514c849760509360113078 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.429106 25925 consensus_queue.cc:1048] T aa59141039de402e9ccce427604ecb31 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.429685 25926 consensus_queue.cc:1048] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.430006 25921 consensus_queue.cc:1048] T 08b8c7c1b50d495f9ed54b8bfa1e984d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.432477 24780 ts_tablet_manager.cc:1916] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.432533 25927 raft_consensus.cc:515] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.432775 25739 leader_election.cc:290] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.430274 25167 consensus_queue.cc:1048] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.430332 23470 leader_election.cc:304] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.433414 25924 raft_consensus.cc:2804] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.433445 25927 leader_election.cc:290] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.435269 25492 ts_tablet_manager.cc:1950] T 857da3642f0846ce92aaed0491e89644 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.436579 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 419) took 4086 ms. Trace:
I20251023 20:15:39.436698 25492 rpcz_store.cc:276] 1023 20:15:35.350243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.350382 (+ 139us) service_pool.cc:225] Handling call
1023 20:15:39.290203 (+3939821us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:39.290213 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.364694 (+ 74481us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.367449 (+ 2755us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.436561 (+ 69112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3939657}
W20251023 20:15:39.436990 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 419) could send its response
I20251023 20:15:39.411391 25276 raft_consensus.cc:493] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.430037 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 390) took 5856 ms. Trace:
I20251023 20:15:39.437026 23787 raft_consensus.cc:385] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.437178 25276 raft_consensus.cc:515] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.437813 25276 leader_election.cc:290] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:39.437316 23787 raft_consensus.cc:740] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.438625 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03f0984da42c49b683870b7c80b224a4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:39.439112 23605 leader_election.cc:343] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.437178 25282 rpcz_store.cc:276] 1023 20:15:33.573149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.573189 (+ 40us) service_pool.cc:225] Handling call
1023 20:15:39.305953 (+5732764us) ts_tablet_manager.cc:1151] Deleting tablet 1e5720c96730425dafb550094cf936cd
1023 20:15:39.306207 (+ 254us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.377146 (+ 70939us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.399661 (+ 22515us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.429920 (+ 30259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4377,"tablet-delete.queue_time_us":5732596}
I20251023 20:15:39.439489 25492 tablet_replica.cc:333] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.439761 25492 raft_consensus.cc:2243] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
W20251023 20:15:39.439750 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 390) could send its response
I20251023 20:15:39.439960 25492 raft_consensus.cc:2272] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.441977 25282 tablet_replica.cc:333] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:39.440141 25924 raft_consensus.cc:697] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.442441 25924 consensus_queue.cc:237] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.440852 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbab01ecb27a426188689408079eaa01" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.441766 23787 consensus_queue.cc:260] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.445691 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_57771b43db0747d68d0dea8e558351e0 [id=c35038526af34fc682bc386c5c80cd2d]
I20251023 20:15:39.441403 23470 leader_election.cc:304] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.444554 25282 raft_consensus.cc:2243] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.446731 25167 raft_consensus.cc:2804] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.446841 25167 raft_consensus.cc:697] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.447041 25167 consensus_queue.cc:237] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.447777 25282 raft_consensus.cc:2272] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.450134 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_00f072ef064344c48ccd7c0982707e50 [id=c4ebced3a66940a6adb331cd4c9ea277]
I20251023 20:15:39.451017 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc]
I20251023 20:15:39.450599 23787 ts_tablet_manager.cc:1434] T cc488f0beec14d0eb869b7474ec16205 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20251023 20:15:39.451568 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_aba7529cde2c47e39145a97c70f643ed [id=c949af1c17974ec991f6bb58c01c4f2b]
I20251023 20:15:39.452131 23790 tablet_bootstrap.cc:492] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:39.452843 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_877d37beafd74b67a50c835a42249829 [id=ca4632b1cfb740fa91e297b0f49f47f3]
I20251023 20:15:39.453377 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d16baf5ed29d4a058c3ef64122fc7309 [id=caafebb3277c4fe3b9188d8ce073ca55]
I20251023 20:15:39.454713 25927 raft_consensus.cc:493] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.454836 25927 raft_consensus.cc:515] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.455166 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_40171fa2448745aaa4c50eac538f6e85 [id=cd774355ec3e41c29e00778e6fd3c23f]
I20251023 20:15:39.455494 25927 leader_election.cc:290] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.454995 23790 ts_tablet_manager.cc:1403] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.172s user 0.002s sys 0.005s
I20251023 20:15:39.455526 23787 tablet_bootstrap.cc:492] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.456661 23470 leader_election.cc:304] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.457064 25739 raft_consensus.cc:2804] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.457120 25921 consensus_queue.cc:1048] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.457928 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_527cb9b18f414f43872b4f9dd4c433e1 [id=cf913261a84244e0a0ef54347a3a548c]
I20251023 20:15:39.457362 25282 ts_tablet_manager.cc:1916] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.457199 25739 raft_consensus.cc:697] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.458990 25739 consensus_queue.cc:237] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.459378 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 [id=d236ba1e610441a8886230b0365ee300]
I20251023 20:15:39.459769 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_ce123b77e83b41c4bf1fc149512c47d7 [id=d3624aa88af54933bf909725fd573d76]
W20251023 20:15:39.460110 23471 leader_election.cc:343] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:39.460438 23471 consensus_peers.cc:597] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a -> Peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Couldn't send request to peer d220aba7c9c0400398ffde917376ee48. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.460856 25466 consensus_queue.cc:1048] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.461440 25784 consensus_queue.cc:1048] T 0eb7935c4cf64e2f802ca12c1f63317b P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.466873 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_f706cbf159424d419faa383ad30ebf77 [id=d399717713214f3dbc8839f2908ba34a]
I20251023 20:15:39.467588 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_43dc7ae468fd42cb867849c762736c66 [id=d53659f9a30541ae913e26824a9ae7d4]
I20251023 20:15:39.468041 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_73bcc4c3d7a44930ba72e4e80ca0eb18 [id=d6f62018af4a4ed58edf91d90e430e01]
I20251023 20:15:39.469025 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_59879b02c1b84af2891ea5c6ebffbca7 [id=dbdd5f771ae84b7498c203156aad3dca]
W20251023 20:15:39.469364 23471 leader_election.cc:343] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.469439 23786 tablet_bootstrap.cc:492] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.469574 23786 ts_tablet_manager.cc:1403] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.000s
I20251023 20:15:39.470176 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066]
I20251023 20:15:39.470618 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac]
I20251023 20:15:39.470664 23786 raft_consensus.cc:359] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.471014 23786 raft_consensus.cc:385] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.471138 23786 raft_consensus.cc:740] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.471185 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d0548a52907a4b1984665988fa00b425 [id=de89b68b2c194fada173999f7a0af035]
I20251023 20:15:39.471643 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525]
I20251023 20:15:39.471485 23786 consensus_queue.cc:260] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.472453 23786 ts_tablet_manager.cc:1434] T 95a06da281194f95b126483314f6debd P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251023 20:15:39.472975 23786 tablet_bootstrap.cc:492] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.473623 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_6b180ae1e48347588da29cf3bd272d3e [id=e145b28bbae940b0bed6cf3cabd00611]
I20251023 20:15:39.476052 23786 tablet_bootstrap.cc:654] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.481987 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b]
I20251023 20:15:39.482704 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c]
I20251023 20:15:39.483157 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7a801afdfd4b4207b0e87c57ee93e01b [id=e74214affb4040149bd7c4db38c87270]
I20251023 20:15:39.483588 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_f06b4a8de0ed4fee83a2d7e2a2eb3823 [id=e95b10e242404ffea012d67b902b54c8]
I20251023 20:15:39.484256 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e]
I20251023 20:15:39.487136 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a]
I20251023 20:15:39.487972 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_69f62f4882e04aa1a32092b258c6acaa [id=ee35b2f67f5f4d9488b1c6484ca4343a]
I20251023 20:15:39.488783 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_1e09690c826a4c028e998750992ec3d4 [id=f120045598ff41b09f32bbffde0e7092]
I20251023 20:15:39.489030 23787 tablet_bootstrap.cc:654] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.490772 25276 consensus_queue.cc:1048] T 5cdde6b251b14df4bf44142ce263b84a P 9bc6522db11e4960a9aad84435371a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.470609 23790 raft_consensus.cc:359] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.497121 23790 raft_consensus.cc:385] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.497201 23790 raft_consensus.cc:740] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.497392 23790 consensus_queue.cc:260] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:39.500129 23604 leader_election.cc:343] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.500268 23604 leader_election.cc:304] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.500746 25276 raft_consensus.cc:2749] T 6313d3307aff4a6880176a06627e86fc P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.500816 25927 raft_consensus.cc:493] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.501005 25927 raft_consensus.cc:515] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:39.501163 23604 leader_election.cc:343] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.501256 23604 leader_election.cc:304] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.501612 25276 raft_consensus.cc:2749] T 38ce80a338d2494aac84fe6344d47e90 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:39.501947 23604 leader_election.cc:343] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.502038 23604 leader_election.cc:304] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.502205 25927 leader_election.cc:290] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.502347 25276 raft_consensus.cc:2749] T 878508d6bf0d4a59b6f64155361b4418 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:39.502686 23604 leader_election.cc:343] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.502775 23604 leader_election.cc:304] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.503090 25276 raft_consensus.cc:2749] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.509465 24297 raft_consensus.cc:493] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.509606 24297 raft_consensus.cc:515] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.510120 24297 leader_election.cc:290] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.511453 23790 ts_tablet_manager.cc:1434] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.055s user 0.000s sys 0.003s
I20251023 20:15:39.512594 23790 tablet_bootstrap.cc:492] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.516711 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a216aeebe649f79b6d873e1c59df37" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.516891 23664 raft_consensus.cc:2468] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.516803 25927 consensus_queue.cc:1048] T 6abfda15e6c242b588d4ed4401008156 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:39.517822 23471 leader_election.cc:343] T b4798f6eea5745ba8a0d4b8cb173cc5c P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.517863 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.517966 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58859629a2f14aacbdb5e2cd0edc27bc" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.518124 23664 raft_consensus.cc:2393] T 58859629a2f14aacbdb5e2cd0edc27bc P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:39.519174 25492 ts_tablet_manager.cc:1916] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:39.520262 23471 leader_election.cc:343] T 9749c6f3fe5545ab93bf49aae97e9818 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.521823 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_d42255cb02014b30a183176f660d20f3 [id=f2b154b7183f4ad0b4fc22a9d5f9620f]
W20251023 20:15:39.521894 23471 leader_election.cc:343] T a9e5dc42c01a4d5e9661f399f793f68f P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.522642 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_49f2383c82e44457827030102d18d5bb [id=f49d21d543f5498580620fda6cfc97e9]
I20251023 20:15:39.523619 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_222ab66f7150462aad2efe8d16484b94 [id=f52ec73cfcfe4b36a08422d69aabf321]
I20251023 20:15:39.524885 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_21c86d62c5c840098934de32762c13cb [id=f55106c921ae47aa8aea61cafa6d309a]
I20251023 20:15:39.526185 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_7ab2fc1342c440f9a0c55b4d52c748f8 [id=f59bbf36d431443a85154ff155cb0e66]
I20251023 20:15:39.528085 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b30e8ca2c96f4063a7f0e8f324c78b12" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.528271 23664 raft_consensus.cc:2468] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.529392 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_9eb13d4c90b44054a9f549bd289a9c6e [id=f97ed62dc77d4f8991056b5268dd88df]
I20251023 20:15:39.531505 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_00e41d94627b471b9753905690b03a49 [id=f9b92376e02047ec9c965526e54739d4]
I20251023 20:15:39.532397 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_eb73fed1f9bb425daff2e3278ad80ece [id=fba1e090c19d406c8c9c4728885f1e28]
I20251023 20:15:39.535578 25775 catalog_manager.cc:679] Loaded metadata for table default.Table_bce343d5f0fc48b5aef18a2668354047 [id=ff726ba9ad7c4b6491970a8c0cd2e42e]
I20251023 20:15:39.542415 23790 tablet_bootstrap.cc:654] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.544416 23528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03f0984da42c49b683870b7c80b224a4" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.546723 25592 raft_consensus.cc:493] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.546867 25592 raft_consensus.cc:515] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.547398 25592 leader_election.cc:290] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.547430 23398 raft_consensus.cc:3060] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.547812 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e5720c96730425dafb550094cf936cd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.548426 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a216aeebe649f79b6d873e1c59df37" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.549018 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b30e8ca2c96f4063a7f0e8f324c78b12" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.549644 23391 raft_consensus.cc:3060] T 757d41e027ba461ea051b38a2d685407 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.550455 25927 raft_consensus.cc:493] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.550590 25927 raft_consensus.cc:515] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.551134 25927 leader_election.cc:290] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.558190 24780 ts_tablet_manager.cc:1929] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.558424 24780 log.cc:1199] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/40c659c273c449939d0dd708d39b393f
I20251023 20:15:39.559118 24780 ts_tablet_manager.cc:1950] T 40c659c273c449939d0dd708d39b393f P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.560477 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 238) took 8509 ms. Trace:
I20251023 20:15:39.562487 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.562712 23664 raft_consensus.cc:2468] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
W20251023 20:15:39.566087 23604 leader_election.cc:343] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.566208 23604 leader_election.cc:304] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.566447 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88f1df552f93427e91ca19c8170d69bd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.566563 25276 raft_consensus.cc:2749] T 03f0984da42c49b683870b7c80b224a4 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.566622 23664 raft_consensus.cc:2468] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.567226 25775 tablet_loader.cc:96] loaded metadata for tablet 0140a384382941a68e9a918d4a82a449 (table default.Table_869b3f73ab4446369b8ab49294b2d32d [id=20d0cb227c7242b29c5ee5423268da12])
I20251023 20:15:39.569237 25775 tablet_loader.cc:96] loaded metadata for tablet 0150ab3778c34c77a7e772dd6e82b32f (table default.Table_0ac0b8ae289f41e391d7d9b51995bd28 [id=0f1dd868b5dc4f62a887203c6a06e113])
I20251023 20:15:39.569898 25775 tablet_loader.cc:96] loaded metadata for tablet 0232c33db59641b3af27bfd15cd7fb81 (table default.Table_d22911da756748fb8939006df660e885 [id=78fe983663c94eabab93ee902bde6998])
I20251023 20:15:39.570127 23786 tablet_bootstrap.cc:492] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.570274 23786 ts_tablet_manager.cc:1403] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.000s
I20251023 20:15:39.570904 25775 tablet_loader.cc:96] loaded metadata for tablet 03f0984da42c49b683870b7c80b224a4 (table default.Table_f06b4a8de0ed4fee83a2d7e2a2eb3823 [id=e95b10e242404ffea012d67b902b54c8])
I20251023 20:15:39.570930 23786 raft_consensus.cc:359] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.571094 23786 raft_consensus.cc:385] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.571156 23786 raft_consensus.cc:740] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.571571 23786 consensus_queue.cc:260] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
W20251023 20:15:39.572005 23470 leader_election.cc:343] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 1e5720c96730425dafb550094cf936cd
I20251023 20:15:39.572175 23470 leader_election.cc:304] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.572374 25921 consensus_queue.cc:1048] T a7f9428277fb4ebf989fed7569dee687 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.572468 25775 tablet_loader.cc:96] loaded metadata for tablet 040a44e272b7410c98c109fbceb19192 (table default.Table_c77f332f1f69484a84091dabddde5d42 [id=70afc97d36fa4bf0a81c6b6611d3975a])
I20251023 20:15:39.572716 25921 raft_consensus.cc:2804] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.572777 25921 raft_consensus.cc:493] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.572830 25921 raft_consensus.cc:3060] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.573063 25775 tablet_loader.cc:96] loaded metadata for tablet 04f7dffd94f3442ea20045a512d7bd65 (table default.Table_b827c36444e34ec6924cfb0ac859b3e1 [id=5bcb88a9719846c78bf5b7609e2bcb37])
I20251023 20:15:39.574656 23787 tablet_bootstrap.cc:492] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.574767 23787 ts_tablet_manager.cc:1403] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.120s user 0.002s sys 0.005s
I20251023 20:15:39.575245 25921 raft_consensus.cc:515] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.575383 23787 raft_consensus.cc:359] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.575513 23787 raft_consensus.cc:385] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.575805 25921 leader_election.cc:290] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.576280 25775 tablet_loader.cc:96] loaded metadata for tablet 08b8c7c1b50d495f9ed54b8bfa1e984d (table default.Table_1b20af8a893a4715affda818cffeca13 [id=201b60a519df46fcb92de45acff546cc])
I20251023 20:15:39.576213 23787 raft_consensus.cc:740] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.576638 23787 consensus_queue.cc:260] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.577925 23787 ts_tablet_manager.cc:1434] T 59c9fe1fc8474c0d92d0964dfd5fd7a3 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:39.578167 25775 tablet_loader.cc:96] loaded metadata for tablet 098af5ca8fa44e939f3b6f7488e09cc3 (table default.Table_c462519a21af4c17bc119f202f50922a [id=7fb99f9e0faa4759ad4c12b5ce1767ea])
I20251023 20:15:39.578331 23790 tablet_bootstrap.cc:492] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:39.578509 23787 tablet_bootstrap.cc:492] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.578611 23790 ts_tablet_manager.cc:1403] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.003s
I20251023 20:15:39.579003 25775 tablet_loader.cc:96] loaded metadata for tablet 0ca01bea2bb146ca8797a9592a840a81 (table default.Table_a3de203f695c4ab09cef39511958ed08 [id=39f5bef03d6c49b0ba2af0223cb9e573])
I20251023 20:15:39.579747 25775 tablet_loader.cc:96] loaded metadata for tablet 0eb7935c4cf64e2f802ca12c1f63317b (table default.Table_8322279a1b514373a3113f54d917fb55 [id=90d69f76e2324f5ea1ea4fee2b9042a6])
I20251023 20:15:39.580173 23790 raft_consensus.cc:359] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.580464 23790 raft_consensus.cc:385] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.580601 23790 raft_consensus.cc:740] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.580875 25775 tablet_loader.cc:96] loaded metadata for tablet 0fe952e2663b426caf6e4d18e98c0312 (table default.Table_b029eee1964748f3bb848b0f6309a357 [id=3d3676ef16f540c190331ed14e25d985])
I20251023 20:15:39.581012 23790 consensus_queue.cc:260] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.581871 25775 tablet_loader.cc:96] loaded metadata for tablet 118be7aa6380469db091d2788472e62b (table default.Table_519888eb86d3490baf10c0d615eea255 [id=672015fbe23a4d5282562f1f4b282485])
I20251023 20:15:39.584818 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.584977 23396 raft_consensus.cc:2468] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.560879 24780 rpcz_store.cc:276] 1023 20:15:31.050897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.050932 (+ 35us) service_pool.cc:225] Handling call
1023 20:15:39.428792 (+8377860us) ts_tablet_manager.cc:1151] Deleting tablet 40c659c273c449939d0dd708d39b393f
1023 20:15:39.428803 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.495020 (+ 66217us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.558080 (+ 63060us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.560457 (+ 2377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8377724}
I20251023 20:15:39.596406 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 239) took 8545 ms. Trace:
I20251023 20:15:39.596812 24780 rpcz_store.cc:276] 1023 20:15:31.051161 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.051196 (+ 35us) service_pool.cc:225] Handling call
1023 20:15:39.596000 (+8544804us) ts_tablet_manager.cc:1151] Deleting tablet 8289ce37fa074891beb584bf2fff9747
1023 20:15:39.596013 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.596292 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8544543}
I20251023 20:15:39.597571 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a216aeebe649f79b6d873e1c59df37" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:39.599164 25282 ts_tablet_manager.cc:1929] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.599256 25282 log.cc:1199] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/ca11ddd5d97544cf916b87fd3dff219d
I20251023 20:15:39.599562 25282 ts_tablet_manager.cc:1950] T ca11ddd5d97544cf916b87fd3dff219d P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:39.599843 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 391) took 6026 ms. Trace:
I20251023 20:15:39.599900 25282 rpcz_store.cc:276] 1023 20:15:33.573823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:33.573895 (+ 72us) service_pool.cc:225] Handling call
1023 20:15:39.441677 (+5867782us) ts_tablet_manager.cc:1151] Deleting tablet ca11ddd5d97544cf916b87fd3dff219d
1023 20:15:39.441690 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.510844 (+ 69154us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.599049 (+ 88205us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.599828 (+ 779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5867385}
W20251023 20:15:39.600107 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 391) could send its response
I20251023 20:15:39.598023 24780 tablet_replica.cc:333] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.600589 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88f1df552f93427e91ca19c8170d69bd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:39.603322 23470 leader_election.cc:304] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.603861 25167 raft_consensus.cc:2804] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.603924 23470 leader_election.cc:304] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.603936 25167 raft_consensus.cc:493] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.604053 25167 raft_consensus.cc:3060] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.604460 25784 raft_consensus.cc:2804] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.604538 25784 raft_consensus.cc:493] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.604615 25784 raft_consensus.cc:3060] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.607003 25784 raft_consensus.cc:515] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.607513 25784 leader_election.cc:290] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.611575 25167 raft_consensus.cc:515] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.612769 25167 leader_election.cc:290] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:39.613466 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 237) could send its response
I20251023 20:15:39.614921 23790 ts_tablet_manager.cc:1434] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251023 20:15:39.616676 24780 raft_consensus.cc:2243] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.618881 23470 leader_election.cc:304] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.618485 23790 tablet_bootstrap.cc:492] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.619154 24780 pending_rounds.cc:70] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Trying to abort 1 pending ops.
I20251023 20:15:39.619639 24780 pending_rounds.cc:77] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214082204318228480 op_type: NO_OP noop_request { }
I20251023 20:15:39.620316 25927 consensus_queue.cc:1048] T 2c40164e687545c5b2d2751baaee4571 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.620828 25927 raft_consensus.cc:2804] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.620911 25927 raft_consensus.cc:493] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.620980 25927 raft_consensus.cc:3060] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.620613 24780 raft_consensus.cc:2889] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251023 20:15:39.622443 24780 raft_consensus.cc:2272] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.623425 25927 raft_consensus.cc:515] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.623951 25927 leader_election.cc:290] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.624655 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.624841 23525 raft_consensus.cc:2393] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:39.585352 25775 tablet_loader.cc:96] loaded metadata for tablet 139c1bb5ffa743d4a62edf5e98932454 (table default.Table_ef7d0ab458b247c5a1998f0538a7b2ea [id=87bf96dc0f3b4ca6867bb468b7501138])
I20251023 20:15:39.626251 23786 ts_tablet_manager.cc:1434] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20251023 20:15:39.626797 25775 tablet_loader.cc:96] loaded metadata for tablet 147781e046fe4feb9815147c774fe210 (table default.Table_50879e22af484b719900c9b120534de0 [id=1245153d2d05472a9f20bc69214ca3ce])
I20251023 20:15:39.627269 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.627485 23665 raft_consensus.cc:2468] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d220aba7c9c0400398ffde917376ee48 in term 0.
I20251023 20:15:39.628159 25775 tablet_loader.cc:96] loaded metadata for tablet 165d62dd42da460faccec706f08272a1 (table default.Table_ce123b77e83b41c4bf1fc149512c47d7 [id=d3624aa88af54933bf909725fd573d76])
I20251023 20:15:39.628441 23786 tablet_bootstrap.cc:492] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.630304 23787 tablet_bootstrap.cc:654] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.631402 25282 tablet_replica.cc:333] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:39.631783 25282 raft_consensus.cc:2243] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.632063 25282 raft_consensus.cc:2272] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.632483 25775 tablet_loader.cc:96] loaded metadata for tablet 1705a87bfa7e4f7caa00b4666fd0514a (table default.Table_c1f42ba6bf654a09ade937991a5f85c0 [id=85e0bb2abec640a28b1cc1806b45e52a])
I20251023 20:15:39.638530 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.640282 24780 ts_tablet_manager.cc:1916] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:39.641952 23471 consensus_peers.cc:597] T ca11ddd5d97544cf916b87fd3dff219d P ea6853efbbcc484eb7529c7e9b37a47a -> Peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Couldn't send request to peer d220aba7c9c0400398ffde917376ee48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca11ddd5d97544cf916b87fd3dff219d. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.643590 25775 tablet_loader.cc:96] loaded metadata for tablet 18939c8f489a4cca833ee3e15b07d71b (table default.Table_fbff76a175884e39a9efd67edd4db37e [id=9ec5b95ed1e340a784ea0a4cbf6e2e7b])
W20251023 20:15:39.644039 23471 leader_election.cc:343] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.644181 23471 leader_election.cc:304] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6853efbbcc484eb7529c7e9b37a47a; no voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48
I20251023 20:15:39.644340 25775 tablet_loader.cc:96] loaded metadata for tablet 19c58a406d564b048fb16a85aa0b754b (table default.Table_ad94afb284cc49b8ab09122184009906 [id=83f2c96018ba47e79c3c7ad4df2852a7])
I20251023 20:15:39.644727 25775 tablet_loader.cc:96] loaded metadata for tablet 19da816a483a4d5b9dc768ef697d54ec (table default.Table_e9fdfa4d73c241e8a7d041b6b327d445 [id=6c150db12c334b46971b50de47e9b641])
I20251023 20:15:39.645037 25924 raft_consensus.cc:2749] T 1e5720c96730425dafb550094cf936cd P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251023 20:15:39.645519 23471 leader_election.cc:343] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:39.646867 23471 leader_election.cc:343] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.648342 25594 raft_consensus.cc:493] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.648473 25594 raft_consensus.cc:515] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.649052 25594 leader_election.cc:290] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.648252 25167 consensus_queue.cc:1048] T fbab01ecb27a426188689408079eaa01 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251023 20:15:39.649799 25921 raft_consensus.cc:493] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.649924 25921 raft_consensus.cc:515] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.650527 25921 leader_election.cc:290] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.633172 25282 ts_tablet_manager.cc:1916] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.642163 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6abfda15e6c242b588d4ed4401008156" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.645364 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3a216aeebe649f79b6d873e1c59df37" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.654100 23665 raft_consensus.cc:3060] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.656359 23786 tablet_bootstrap.cc:654] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.647258 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.657073 23668 raft_consensus.cc:3060] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.657883 25492 ts_tablet_manager.cc:1929] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:39.660282 23668 raft_consensus.cc:2468] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.648897 23660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88f1df552f93427e91ca19c8170d69bd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.647732 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b30e8ca2c96f4063a7f0e8f324c78b12" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.661036 23660 raft_consensus.cc:3060] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.661098 25492 log.cc:1199] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/39a62c97e0a3459e91c24c1c2f92389a
I20251023 20:15:39.661103 23663 raft_consensus.cc:3060] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.661396 25492 ts_tablet_manager.cc:1950] T 39a62c97e0a3459e91c24c1c2f92389a P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.662081 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 421) took 4311 ms. Trace:
I20251023 20:15:39.662153 25492 rpcz_store.cc:276] 1023 20:15:35.350424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.350577 (+ 153us) service_pool.cc:225] Handling call
1023 20:15:39.439441 (+4088864us) ts_tablet_manager.cc:1151] Deleting tablet 39a62c97e0a3459e91c24c1c2f92389a
1023 20:15:39.439452 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.522484 (+ 83032us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.657780 (+135296us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.662061 (+ 4281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4088722}
W20251023 20:15:39.662364 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 421) could send its response
I20251023 20:15:39.663628 23663 raft_consensus.cc:2468] T b30e8ca2c96f4063a7f0e8f324c78b12 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.664446 23660 raft_consensus.cc:2468] T 88f1df552f93427e91ca19c8170d69bd P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.664889 23665 raft_consensus.cc:2468] T b3a216aeebe649f79b6d873e1c59df37 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.669304 25492 tablet_replica.cc:333] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.670470 25492 raft_consensus.cc:2243] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.673429 23786 tablet_bootstrap.cc:492] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.674526 23786 ts_tablet_manager.cc:1403] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.004s
I20251023 20:15:39.675441 23786 raft_consensus.cc:359] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.675769 23786 raft_consensus.cc:385] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.675933 23786 raft_consensus.cc:740] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.674798 25492 raft_consensus.cc:2272] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.676901 25492 ts_tablet_manager.cc:1916] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.679946 23786 consensus_queue.cc:260] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.681522 25775 tablet_loader.cc:96] loaded metadata for tablet 1b220ceff0924b26ae8a54dfb8705456 (table default.Table_b9e2bcb23af94936a9fbfd0817d9a51e [id=3528c0390b7240e8a565945035555dfa])
I20251023 20:15:39.681838 23470 leader_election.cc:304] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.682808 23786 ts_tablet_manager.cc:1434] T 118be7aa6380469db091d2788472e62b P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251023 20:15:39.684010 23786 tablet_bootstrap.cc:492] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.685303 23787 tablet_bootstrap.cc:492] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.685442 23787 ts_tablet_manager.cc:1403] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.000s
I20251023 20:15:39.686046 23787 raft_consensus.cc:359] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.686198 23787 raft_consensus.cc:385] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.686241 23787 raft_consensus.cc:740] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.686368 23786 tablet_bootstrap.cc:654] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.686581 23787 consensus_queue.cc:260] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.688195 23787 ts_tablet_manager.cc:1434] T 6ecfa16e75f543fab12fa0bac4d9e3e8 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251023 20:15:39.689584 23787 tablet_bootstrap.cc:492] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.692550 25492 ts_tablet_manager.cc:1929] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.692688 25492 log.cc:1199] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/370a790a053446b2b06314c7db242270
I20251023 20:15:39.693039 25492 ts_tablet_manager.cc:1950] T 370a790a053446b2b06314c7db242270 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
W20251023 20:15:39.695611 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 238) could send its response
W20251023 20:15:39.696110 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 239) could send its response
I20251023 20:15:39.699256 23790 tablet_bootstrap.cc:654] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.705822 25927 raft_consensus.cc:2804] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.705976 25927 raft_consensus.cc:697] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.706236 25927 consensus_queue.cc:237] T dd07383242b74637bac46609f56c7e4d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.706521 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 423) took 4355 ms. Trace:
I20251023 20:15:39.706619 25492 rpcz_store.cc:276] 1023 20:15:35.350628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.350741 (+ 113us) service_pool.cc:225] Handling call
1023 20:15:39.669254 (+4318513us) ts_tablet_manager.cc:1151] Deleting tablet 370a790a053446b2b06314c7db242270
1023 20:15:39.669263 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.689419 (+ 20156us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.692458 (+ 3039us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.706498 (+ 14040us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12640,"tablet-delete.queue_time_us":4318372}
I20251023 20:15:39.706934 25492 tablet_replica.cc:333] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.707106 25492 raft_consensus.cc:2243] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.707226 25492 raft_consensus.cc:2272] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.707715 25775 tablet_loader.cc:96] loaded metadata for tablet 1e343a7d39cf4329b99c9f9ffd8cefb2 (table default.Table_322349f14bf94a7cbd9cb6cb92793905 [id=9ae6a893fdef4f328cbfd0ee2d57f5a1])
I20251023 20:15:39.708614 25925 consensus_queue.cc:1048] T 757d41e027ba461ea051b38a2d685407 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:39.709338 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 423) could send its response
I20251023 20:15:39.710630 25492 ts_tablet_manager.cc:1916] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.712242 23787 tablet_bootstrap.cc:654] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.715924 25276 raft_consensus.cc:493] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.716055 25276 raft_consensus.cc:515] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.716655 25276 leader_election.cc:290] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:39.717382 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f98f490c2caa49b8ae08241e5bb4df9b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:39.717959 23605 leader_election.cc:343] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
W20251023 20:15:39.719942 23333 leader_election.cc:343] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: fbab01ecb27a426188689408079eaa01
I20251023 20:15:39.722168 23333 leader_election.cc:304] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, d220aba7c9c0400398ffde917376ee48; no voters:
I20251023 20:15:39.722935 25276 raft_consensus.cc:493] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.723073 25276 raft_consensus.cc:515] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.723635 25276 leader_election.cc:290] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:39.724339 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e289b51f48a942fe91b95f528130f89b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:39.724843 23605 leader_election.cc:343] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.721752 25492 ts_tablet_manager.cc:1929] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:39.727530 25492 log.cc:1199] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/8317a54a5f7b4afe8da9d336268efe43
I20251023 20:15:39.728065 25492 ts_tablet_manager.cc:1950] T 8317a54a5f7b4afe8da9d336268efe43 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.729247 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 422) took 4378 ms. Trace:
I20251023 20:15:39.732326 25492 rpcz_store.cc:276] 1023 20:15:35.350507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.350838 (+ 331us) service_pool.cc:225] Handling call
1023 20:15:39.706886 (+4356048us) ts_tablet_manager.cc:1151] Deleting tablet 8317a54a5f7b4afe8da9d336268efe43
1023 20:15:39.706895 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.713680 (+ 6785us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.720735 (+ 7055us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.729226 (+ 8491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4355920}
W20251023 20:15:39.732801 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 422) could send its response
I20251023 20:15:39.733136 25492 tablet_replica.cc:333] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.733526 25492 raft_consensus.cc:2243] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.733263 25282 ts_tablet_manager.cc:1929] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.733968 25492 raft_consensus.cc:2272] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.734325 25282 log.cc:1199] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/00ec606cbf834ccd8762e034b520e58e
I20251023 20:15:39.735491 25492 ts_tablet_manager.cc:1916] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.738121 23790 tablet_bootstrap.cc:492] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:39.738263 23790 ts_tablet_manager.cc:1403] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.004s
I20251023 20:15:39.738114 25282 ts_tablet_manager.cc:1950] T 00ec606cbf834ccd8762e034b520e58e P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:39.738969 23790 raft_consensus.cc:359] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.739142 23790 raft_consensus.cc:385] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.739192 23790 raft_consensus.cc:740] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.739367 23790 consensus_queue.cc:260] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.747443 25775 tablet_loader.cc:96] loaded metadata for tablet 1edaed8c82da4e36a312477b15017460 (table default.Table_00e41d94627b471b9753905690b03a49 [id=f9b92376e02047ec9c965526e54739d4])
I20251023 20:15:39.748857 25775 tablet_loader.cc:96] loaded metadata for tablet 215e21ca449c481d8716783a65105101 (table default.Table_efcc5e094c4349d686c5e8af614c2c5c [id=b68a3df9df3b49379181ef2b7573e973])
I20251023 20:15:39.750218 25726 raft_consensus.cc:2804] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.750340 25726 raft_consensus.cc:493] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.750433 25726 raft_consensus.cc:3060] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.752913 25726 raft_consensus.cc:515] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.753546 25726 leader_election.cc:290] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 election: Requested vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.753893 25492 ts_tablet_manager.cc:1929] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.754110 25492 log.cc:1199] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/1822fdf1f5f44dd4a742aa3c2f8443cf
I20251023 20:15:39.754480 25492 ts_tablet_manager.cc:1950] T 1822fdf1f5f44dd4a742aa3c2f8443cf P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.754184 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 393) took 4803 ms. Trace:
I20251023 20:15:39.755339 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 424) took 4404 ms. Trace:
I20251023 20:15:39.755561 25492 rpcz_store.cc:276] 1023 20:15:35.350714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.350905 (+ 191us) service_pool.cc:225] Handling call
1023 20:15:39.733072 (+4382167us) ts_tablet_manager.cc:1151] Deleting tablet 1822fdf1f5f44dd4a742aa3c2f8443cf
1023 20:15:39.733097 (+ 25us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.746281 (+ 13184us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.753779 (+ 7498us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.755321 (+ 1542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4382047}
W20251023 20:15:39.755847 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 424) could send its response
I20251023 20:15:39.755477 25997 raft_consensus.cc:493] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d220aba7c9c0400398ffde917376ee48)
I20251023 20:15:39.756064 25997 raft_consensus.cc:515] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.756647 25997 leader_election.cc:290] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.757584 23038 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
/tmp/dist-test-taskvdN7pO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.127.188:41417
--webserver_interface=127.22.127.188
--webserver_port=41587
--builtin_ntp_servers=127.22.127.148:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.127.190:34331,127.22.127.189:33509,127.22.127.188:41417
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251023 20:15:39.756949 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 425) took 4406 ms. Trace:
I20251023 20:15:39.758095 25492 rpcz_store.cc:276] 1023 20:15:35.350797 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.350998 (+ 201us) service_pool.cc:225] Handling call
1023 20:15:39.756820 (+4405822us) ts_tablet_manager.cc:1151] Deleting tablet 1e19859a65a144598e1612be89daa3a1
1023 20:15:39.756832 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.756932 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4405704}
I20251023 20:15:39.755061 25282 rpcz_store.cc:276] 1023 20:15:34.950304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:34.950363 (+ 59us) service_pool.cc:225] Handling call
1023 20:15:39.600406 (+4650043us) ts_tablet_manager.cc:1151] Deleting tablet 00ec606cbf834ccd8762e034b520e58e
1023 20:15:39.600417 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.691058 (+ 90641us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.732372 (+ 41314us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.754158 (+ 21786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4649792}
W20251023 20:15:39.758291 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 425) could send its response
I20251023 20:15:39.758531 25492 tablet_replica.cc:333] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
W20251023 20:15:39.758600 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 393) could send its response
I20251023 20:15:39.758709 25492 raft_consensus.cc:2243] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.760528 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 394) took 4464 ms. Trace:
I20251023 20:15:39.761348 25282 rpcz_store.cc:276] 1023 20:15:35.296050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.296158 (+ 108us) service_pool.cc:225] Handling call
1023 20:15:39.759700 (+4463542us) ts_tablet_manager.cc:1151] Deleting tablet e194d03a6aa946fbb2c79d91b196440b
1023 20:15:39.759712 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.760505 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4463112}
I20251023 20:15:39.762523 23786 tablet_bootstrap.cc:492] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.762648 23786 ts_tablet_manager.cc:1403] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.004s
W20251023 20:15:39.762756 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 394) could send its response
I20251023 20:15:39.763345 23786 raft_consensus.cc:359] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.763495 23786 raft_consensus.cc:385] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.763669 23786 raft_consensus.cc:740] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.763895 23786 consensus_queue.cc:260] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.764386 23786 ts_tablet_manager.cc:1434] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:39.764199 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 395) took 4468 ms. Trace:
I20251023 20:15:39.764868 23786 tablet_bootstrap.cc:492] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.764981 25282 rpcz_store.cc:276] 1023 20:15:35.296176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.296393 (+ 217us) service_pool.cc:225] Handling call
1023 20:15:39.763638 (+4467245us) ts_tablet_manager.cc:1151] Deleting tablet 7e36bb5e61d34b35a373bd6a51a7ac5e
1023 20:15:39.763653 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.764182 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4466874}
I20251023 20:15:39.705986 23470 leader_election.cc:304] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:39.766364 23470 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.127.131:46211, user_credentials={real_user=slave}} blocked reactor thread for 60431.5us
W20251023 20:15:39.766409 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 395) could send its response
I20251023 20:15:39.766551 23470 leader_election.cc:304] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.766923 25997 raft_consensus.cc:2804] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.766953 23470 leader_election.cc:304] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.767038 25997 raft_consensus.cc:697] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.767177 25926 raft_consensus.cc:2804] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.767277 25926 raft_consensus.cc:697] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.767247 25997 consensus_queue.cc:237] T b30e8ca2c96f4063a7f0e8f324c78b12 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.767505 25926 consensus_queue.cc:237] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.768859 25739 raft_consensus.cc:2804] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.768998 25739 raft_consensus.cc:697] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.769213 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.768747 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 396) took 4472 ms. Trace:
I20251023 20:15:39.769388 23665 raft_consensus.cc:2468] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.769289 25739 consensus_queue.cc:237] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.769595 25282 rpcz_store.cc:276] 1023 20:15:35.296259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.296570 (+ 311us) service_pool.cc:225] Handling call
1023 20:15:39.767280 (+4470710us) ts_tablet_manager.cc:1151] Deleting tablet 26a680ad3aed487593bd1b63004dbbfc
1023 20:15:39.767292 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.768728 (+ 1436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4470351}
W20251023 20:15:39.772434 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 396) could send its response
I20251023 20:15:39.774848 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 397) took 4478 ms. Trace:
I20251023 20:15:39.775460 25282 rpcz_store.cc:276] 1023 20:15:35.296333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.296719 (+ 386us) service_pool.cc:225] Handling call
1023 20:15:39.773725 (+4477006us) ts_tablet_manager.cc:1151] Deleting tablet 40c659c273c449939d0dd708d39b393f
1023 20:15:39.773738 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.774587 (+ 849us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4476652}
W20251023 20:15:39.776775 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 397) could send its response
I20251023 20:15:39.778524 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 398) took 4481 ms. Trace:
I20251023 20:15:39.779201 25282 rpcz_store.cc:276] 1023 20:15:35.296432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.296864 (+ 432us) service_pool.cc:225] Handling call
1023 20:15:39.777532 (+4480668us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:39.777854 (+ 322us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.778272 (+ 418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4480554}
W20251023 20:15:39.780459 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 398) could send its response
I20251023 20:15:39.781680 25282 tablet_replica.cc:333] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:39.783596 25282 raft_consensus.cc:2243] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.785327 25282 raft_consensus.cc:2272] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251023 20:15:39.791263 23471 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.127.129:34961, user_credentials={real_user=slave}} blocked reactor thread for 83388us
I20251023 20:15:39.790863 25282 ts_tablet_manager.cc:1916] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.794430 23786 tablet_bootstrap.cc:654] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:39.796366 23471 leader_election.cc:343] T b3a216aeebe649f79b6d873e1c59df37 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.796898 25492 raft_consensus.cc:2272] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.804790 25492 ts_tablet_manager.cc:1916] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.807765 25726 consensus_queue.cc:1048] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251023 20:15:39.810321 23333 consensus_peers.cc:597] T 00ec606cbf834ccd8762e034b520e58e P d220aba7c9c0400398ffde917376ee48 -> Peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Couldn't send request to peer 9bc6522db11e4960a9aad84435371a32. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.810905 25997 raft_consensus.cc:493] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.811017 25997 raft_consensus.cc:515] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.811183 25726 consensus_queue.cc:1048] T cf38936bb12846bd81a2c79ffbd18ccb P d220aba7c9c0400398ffde917376ee48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.811439 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f98f490c2caa49b8ae08241e5bb4df9b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.811513 25997 leader_election.cc:290] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.811865 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e289b51f48a942fe91b95f528130f89b" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.812194 23333 leader_election.cc:304] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.812580 25726 raft_consensus.cc:2749] T fbab01ecb27a426188689408079eaa01 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.814118 25492 ts_tablet_manager.cc:1929] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.814244 25492 log.cc:1199] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/a841ffdc6cdd432dac7afe2e2b53f224
I20251023 20:15:39.814600 25492 ts_tablet_manager.cc:1950] T a841ffdc6cdd432dac7afe2e2b53f224 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.814915 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 420) took 4464 ms. Trace:
W20251023 20:15:39.815085 23604 leader_election.cc:343] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.815184 23604 leader_election.cc:304] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.815268 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb2bf696d2f4b639dfb0eff2a65e6f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.815404 23664 raft_consensus.cc:2468] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.815018 25492 rpcz_store.cc:276] 1023 20:15:35.350341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.351268 (+ 927us) service_pool.cc:225] Handling call
1023 20:15:39.758486 (+4407218us) ts_tablet_manager.cc:1151] Deleting tablet a841ffdc6cdd432dac7afe2e2b53f224
1023 20:15:39.758496 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.807685 (+ 49189us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.814027 (+ 6342us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.814900 (+ 873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4407075}
W20251023 20:15:39.815550 23604 leader_election.cc:343] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.815630 23604 leader_election.cc:304] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.815683 23470 leader_election.cc:304] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
W20251023 20:15:39.815802 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 420) could send its response
I20251023 20:15:39.815858 25854 raft_consensus.cc:2749] T f98f490c2caa49b8ae08241e5bb4df9b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.815954 25167 raft_consensus.cc:2804] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.815948 25276 raft_consensus.cc:2749] T e289b51f48a942fe91b95f528130f89b P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.816008 25167 raft_consensus.cc:493] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.816068 25167 raft_consensus.cc:3060] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
W20251023 20:15:39.816048 23470 leader_election.cc:343] T d62dbf044acd4459b26d2e7e50f6206e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: d62dbf044acd4459b26d2e7e50f6206e
I20251023 20:15:39.818400 25167 raft_consensus.cc:515] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.818941 25167 leader_election.cc:290] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.819244 25492 tablet_replica.cc:333] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.819438 25492 raft_consensus.cc:2243] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.819584 25492 raft_consensus.cc:2272] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.820073 25492 ts_tablet_manager.cc:1916] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.823186 23470 leader_election.cc:304] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.825502 25775 tablet_loader.cc:96] loaded metadata for tablet 2202632a57514c849760509360113078 (table default.Table_d4d5c2fd468445729fb6a7987712bd39 [id=de3be1729400426bb9957ccede88ceac])
I20251023 20:15:39.826149 25775 tablet_loader.cc:96] loaded metadata for tablet 225ce6aa52c54f76bd56247f01219dbc (table default.Table_7824bcf5ff034be78ed7dee87e4cd78d [id=44239d72fdcd48258921b9490c822137])
I20251023 20:15:39.826506 25775 tablet_loader.cc:96] loaded metadata for tablet 22b7598fe6734d9098fcaa0a0f4bdc7b (table default.Table_cae8a4c4d4b04f7bbc7fb29d0f7f179b [id=58cb342d7d6142858153064d66e3712c])
I20251023 20:15:39.827096 25775 tablet_loader.cc:96] loaded metadata for tablet 233d4e7864804665a61a9acca305360b (table default.Table_bdcbe2f8128a4a89b305439e5123c2fe [id=452652d5f027441cb6f821e5d1db0c03])
I20251023 20:15:39.827456 25775 tablet_loader.cc:96] loaded metadata for tablet 244fa327fad74aad9e57a7fbcbf908bb (table default.Table_7366a659fef84d17925d61af9a4d456c [id=910ad9776933488eb34806b4aaff92d8])
I20251023 20:15:39.830358 25997 raft_consensus.cc:2804] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.830444 25997 raft_consensus.cc:493] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.830498 25997 raft_consensus.cc:3060] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.831563 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.831727 23666 raft_consensus.cc:3060] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.831919 23787 tablet_bootstrap.cc:492] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.832041 23787 ts_tablet_manager.cc:1403] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.143s user 0.004s sys 0.000s
I20251023 20:15:39.832696 23787 raft_consensus.cc:359] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.832800 25997 raft_consensus.cc:515] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.832876 23787 raft_consensus.cc:385] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.832921 23787 raft_consensus.cc:740] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.833134 23787 consensus_queue.cc:260] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.833580 23787 ts_tablet_manager.cc:1434] T b74d2e6d360c4200ae8740dddc206401 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251023 20:15:39.833813 25997 leader_election.cc:290] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.833885 23787 tablet_bootstrap.cc:492] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.834615 23666 raft_consensus.cc:2468] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.835367 23787 tablet_bootstrap.cc:654] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.835496 23790 ts_tablet_manager.cc:1434] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.097s user 0.002s sys 0.000s
I20251023 20:15:39.835865 23790 tablet_bootstrap.cc:492] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.838613 25492 ts_tablet_manager.cc:1929] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:39.838759 25492 log.cc:1199] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/9a4f012469fa4e8e827e485287ad76dd
I20251023 20:15:39.839075 25492 ts_tablet_manager.cc:1950] T 9a4f012469fa4e8e827e485287ad76dd P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.839841 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 426) took 4488 ms. Trace:
I20251023 20:15:39.839956 25492 rpcz_store.cc:276] 1023 20:15:35.351231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.351439 (+ 208us) service_pool.cc:225] Handling call
1023 20:15:39.819199 (+4467760us) ts_tablet_manager.cc:1151] Deleting tablet 9a4f012469fa4e8e827e485287ad76dd
1023 20:15:39.819208 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.835314 (+ 16106us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.838506 (+ 3192us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.839821 (+ 1315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4467639}
I20251023 20:15:39.840288 24780 ts_tablet_manager.cc:1929] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251023 20:15:39.840476 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 426) could send its response
I20251023 20:15:39.840534 25492 tablet_replica.cc:333] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:39.840420 24780 log.cc:1199] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/9a6a2d2b25824dbdafaf2a9cabb0f061
I20251023 20:15:39.840955 25492 raft_consensus.cc:2243] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:39.841615 23787 tablet_bootstrap.cc:492] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.842124 23787 ts_tablet_manager.cc:1403] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20251023 20:15:39.842145 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb2bf696d2f4b639dfb0eff2a65e6f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.842458 23666 raft_consensus.cc:3060] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.841495 24780 ts_tablet_manager.cc:1950] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:39.844385 23790 tablet_bootstrap.cc:654] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
W20251023 20:15:39.846838 23333 consensus_peers.cc:597] T 26a680ad3aed487593bd1b63004dbbfc P d220aba7c9c0400398ffde917376ee48 -> Peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Couldn't send request to peer ea6853efbbcc484eb7529c7e9b37a47a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26a680ad3aed487593bd1b63004dbbfc. This is attempt 1: this message will repeat every 5th retry.
I20251023 20:15:39.847347 23787 raft_consensus.cc:359] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.847496 23787 raft_consensus.cc:385] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.847548 23787 raft_consensus.cc:740] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.847760 23787 consensus_queue.cc:260] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.848235 23787 ts_tablet_manager.cc:1434] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20251023 20:15:39.848491 23787 tablet_bootstrap.cc:492] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.848444 23666 raft_consensus.cc:2468] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:39.850055 23787 tablet_bootstrap.cc:654] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.841892 23470 leader_election.cc:304] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.851073 25492 raft_consensus.cc:2272] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.851418 23470 leader_election.cc:304] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.851682 26005 raft_consensus.cc:2804] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.851787 26005 raft_consensus.cc:697] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.851856 25997 raft_consensus.cc:2804] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:39.851966 25997 raft_consensus.cc:697] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:39.852037 26005 consensus_queue.cc:237] T aec7c5ea622949aca6c3dbe0e5f8e81e P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.852162 25997 consensus_queue.cc:237] T 3bb2bf696d2f4b639dfb0eff2a65e6f8 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.858950 25492 ts_tablet_manager.cc:1916] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.863256 23333 leader_election.cc:304] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.866719 25726 raft_consensus.cc:3060] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.869422 25726 raft_consensus.cc:2749] T 58859629a2f14aacbdb5e2cd0edc27bc P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:39.876533 23790 tablet_bootstrap.cc:492] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:39.878753 23790 ts_tablet_manager.cc:1403] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20251023 20:15:39.879495 25775 tablet_loader.cc:96] loaded metadata for tablet 255521d943d34713aaf773b15f9168c6 (table default.Table_e2bc01c543da4117a80827430a7d66a5 [id=a48bcc6da6504ccb941c18dd5e289c75])
I20251023 20:15:39.880170 25775 tablet_loader.cc:96] loaded metadata for tablet 261834d00bd54fbbb3ea0b3cd44bc393 (table default.Table_dce798c03c814b1899812dc8c06f4452 [id=6f9fc2e5ee6146e3a074b5fbd14314f1])
I20251023 20:15:39.881127 25775 tablet_loader.cc:96] loaded metadata for tablet 27b593b020b34a969f77d2f68178e89d (table default.Table_baf8abd9a132412a8b53ac8d9e044dcf [id=51ea9f032fdd40bb9f0ea42bb80c5cbf])
I20251023 20:15:39.891327 23790 raft_consensus.cc:359] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.892874 23790 raft_consensus.cc:385] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.904264 23786 tablet_bootstrap.cc:492] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.904507 23786 ts_tablet_manager.cc:1403] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.140s user 0.022s sys 0.010s
I20251023 20:15:39.908290 25997 consensus_queue.cc:1048] T 3e7909dba52748bdb78e08fa4b8a2d47 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.910059 23786 raft_consensus.cc:359] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.910224 23786 raft_consensus.cc:385] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.910275 23786 raft_consensus.cc:740] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.911296 25492 ts_tablet_manager.cc:1929] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:39.913232 25775 tablet_loader.cc:96] loaded metadata for tablet 29c2c976d6a3440394f478a31abc7134 (table default.Table_43dc7ae468fd42cb867849c762736c66 [id=d53659f9a30541ae913e26824a9ae7d4])
I20251023 20:15:39.916476 23787 tablet_bootstrap.cc:492] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.916625 23787 ts_tablet_manager.cc:1403] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.000s
I20251023 20:15:39.917339 23787 raft_consensus.cc:359] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.917505 23787 raft_consensus.cc:385] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.917564 23787 raft_consensus.cc:740] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.917052 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 240) took 8865 ms. Trace:
I20251023 20:15:39.917742 23787 consensus_queue.cc:260] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.917754 24780 rpcz_store.cc:276] 1023 20:15:31.051461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.051496 (+ 35us) service_pool.cc:225] Handling call
1023 20:15:39.597869 (+8546373us) ts_tablet_manager.cc:1151] Deleting tablet 9a6a2d2b25824dbdafaf2a9cabb0f061
1023 20:15:39.597880 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.617800 (+ 19920us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x616000173e54. stack: 00007f0b535193e6 00007f0b535191a0 00007f0b524f729d 0000000000983974 000000000098393d 000000000098330a 00007f0b5f3c2f20 00007f0b607104e3 00007f0b632d0e07 00007f0b63303e54 00007f0b632e48f0 00007f0b632e467d 0000000000a880ec 00007f0b5355f76f 00007f0b53562a3b 00007f0b535628b8
1023 20:15:39.703885 (+ 86085us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.840183 (+136298us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.917029 (+ 76846us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17819,"spinlock_wait_cycles":31200000,"tablet-delete.queue_time_us":8546109}
I20251023 20:15:39.918479 23787 ts_tablet_manager.cc:1434] T 94b4838644a9403693572350b1bdd0e2 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:39.919220 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 241) took 8867 ms. Trace:
I20251023 20:15:39.919310 24780 rpcz_store.cc:276] 1023 20:15:31.051812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.051850 (+ 38us) service_pool.cc:225] Handling call
1023 20:15:39.918112 (+8866262us) ts_tablet_manager.cc:1151] Deleting tablet ae388da9e4f94ba5af82a67088ab7097
1023 20:15:39.918124 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.919199 (+ 1075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8865552}
I20251023 20:15:39.919582 24780 tablet_replica.cc:333] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:39.919785 24780 raft_consensus.cc:2243] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.919941 24780 raft_consensus.cc:2272] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.920575 24780 ts_tablet_manager.cc:1916] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.921258 23787 tablet_bootstrap.cc:492] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.910434 23786 consensus_queue.cc:260] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.922042 25492 log.cc:1199] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/f9bd2b109bfa4804a372965e60ba6e85
I20251023 20:15:39.922289 25282 ts_tablet_manager.cc:1929] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:39.923951 23787 tablet_bootstrap.cc:654] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.922695 23786 ts_tablet_manager.cc:1434] T 2e735945647148478201eeffa1d03632 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251023 20:15:39.924358 25492 ts_tablet_manager.cc:1950] T f9bd2b109bfa4804a372965e60ba6e85 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:39.924607 25775 tablet_loader.cc:96] loaded metadata for tablet 29f5849dd3a949a98a415ef9e8076358 (table default.Table_7417d01d91d44665a6f021a647db582c [id=580458858c744a5f98da3fa134aea223])
I20251023 20:15:39.925163 25775 tablet_loader.cc:96] loaded metadata for tablet 2a1bf4f01fbc4645bcf8392f56f853b2 (table default.Table_f021ca39359245598311be2bc68bbbcc [id=7a4098cc9d4e49a39ef792883f85d564])
I20251023 20:15:39.893467 23790 raft_consensus.cc:740] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.926869 25775 tablet_loader.cc:96] loaded metadata for tablet 2c40164e687545c5b2d2751baaee4571 (table default.Table_7cbd6649f17d4bbfbb261c0a4bee7dc3 [id=6defcb8665844b1d958c71d4fc34263a])
I20251023 20:15:39.923545 25282 log.cc:1199] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/7f2a4f1d52984af58d6286671389b6a9
I20251023 20:15:39.928201 25775 tablet_loader.cc:96] loaded metadata for tablet 2d8ded8261ed406d88df394420730a31 (table default.Table_b3b6a3c4f36f4f888a6b5c483002e6ba [id=52a26bb4b9384c829830c1fe988d04be])
I20251023 20:15:39.928092 23333 leader_election.cc:304] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d220aba7c9c0400398ffde917376ee48; no voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:39.928391 23790 consensus_queue.cc:260] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.930619 23786 tablet_bootstrap.cc:492] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:39.930420 25594 raft_consensus.cc:2749] T 6abfda15e6c242b588d4ed4401008156 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251023 20:15:39.932682 25594 raft_consensus.cc:493] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.932797 25594 raft_consensus.cc:515] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.933367 25594 leader_election.cc:290] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:39.935750 23471 leader_election.cc:343] T 88f1df552f93427e91ca19c8170d69bd P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:39.936398 23786 tablet_bootstrap.cc:654] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.936909 23790 ts_tablet_manager.cc:1434] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.057s user 0.003s sys 0.000s
I20251023 20:15:39.942571 23787 tablet_bootstrap.cc:492] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.942726 23787 ts_tablet_manager.cc:1403] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251023 20:15:39.943006 25775 tablet_loader.cc:96] loaded metadata for tablet 2e735945647148478201eeffa1d03632 (table default.Table_aba7529cde2c47e39145a97c70f643ed [id=c949af1c17974ec991f6bb58c01c4f2b])
I20251023 20:15:39.943522 23787 raft_consensus.cc:359] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.943699 23787 raft_consensus.cc:385] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.943758 23787 raft_consensus.cc:740] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.943979 23787 consensus_queue.cc:260] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.944550 25775 tablet_loader.cc:96] loaded metadata for tablet 2fea0afc8f63497d8b5cdf2352a20267 (table default.Table_a9a54be33bfd4ab79fbc7c01381625b6 [id=991a6048a0894f9f95f52a8f0b881db6])
I20251023 20:15:39.945571 25997 consensus_queue.cc:1048] T 8a45d9b3031b46d390e0a1010489c718 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:39.948077 25775 tablet_loader.cc:96] loaded metadata for tablet 31df5c2c40de4d88b6ab1567b84ce4ef (table default.Table_20937abab3a8462cb783c7a4f34a9600 [id=afc43a90036d4b9cb050f975473f2e56])
I20251023 20:15:39.948959 25775 tablet_loader.cc:96] loaded metadata for tablet 348fc49768fb4bb4bfc6a4b5beedcdd5 (table default.Table_03569553d00b42c890bd05b75ff28289 [id=a5b078d079dc454ab853c988889beb21])
I20251023 20:15:39.944453 23787 ts_tablet_manager.cc:1434] T fe49ee5aaf784ef7a9078ff504ddee70 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:39.949424 23787 tablet_bootstrap.cc:492] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.951079 23787 tablet_bootstrap.cc:654] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.944978 25282 ts_tablet_manager.cc:1950] T 7f2a4f1d52984af58d6286671389b6a9 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:39.953581 25775 tablet_loader.cc:96] loaded metadata for tablet 34a738510f514b088469de3d8a39ca2c (table default.Table_8aa752a341ce41b896d76afaf6a0d4f8 [id=07cb3d69214745d3aa5003e92b2a09c3])
I20251023 20:15:39.954547 25775 tablet_loader.cc:96] loaded metadata for tablet 34f22a64e5754eb7b78357323cd27712 (table default.Table_e6fec6ab98474529a2fdf84e3d3cdd2d [id=183b38fa40b949b8b4ca994b1a5efdd1])
W20251023 20:15:39.955274 24840 leader_election.cc:336] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6545140d94b7485aa81c51ad233355cd (127.22.127.190:34331): Timed out: RequestConsensusVote RPC to 127.22.127.190:34331 timed out after 0.621s (SENT)
I20251023 20:15:39.955446 24840 leader_election.cc:304] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae5a310b4a514c0f812bda133c756062; no voters: 6545140d94b7485aa81c51ad233355cd, 690ffde75cd745b6a87925967d54471a
I20251023 20:15:39.961542 25775 tablet_loader.cc:96] loaded metadata for tablet 37c8403b53f04a8e97ffe058e59b425c (table default.Table_191b94017cca4cfb946f38ab19ebfe20 [id=330c829b79fd4cce881fc565a3463863])
I20251023 20:15:39.961684 23787 tablet_bootstrap.cc:492] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.961830 23787 ts_tablet_manager.cc:1403] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20251023 20:15:39.962538 23787 raft_consensus.cc:359] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.962707 23787 raft_consensus.cc:385] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.962765 23787 raft_consensus.cc:740] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.963193 23787 consensus_queue.cc:260] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.963697 23787 ts_tablet_manager.cc:1434] T 886838284a0548f6bb089f62c686165e P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251023 20:15:39.964056 23787 tablet_bootstrap.cc:492] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.965737 23787 tablet_bootstrap.cc:654] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.967438 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 399) took 4669 ms. Trace:
I20251023 20:15:39.967746 25282 rpcz_store.cc:276] 1023 20:15:35.297672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.297718 (+ 46us) service_pool.cc:225] Handling call
1023 20:15:39.781387 (+4483669us) ts_tablet_manager.cc:1151] Deleting tablet 7f2a4f1d52984af58d6286671389b6a9
1023 20:15:39.781398 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.871765 (+ 90367us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.921522 (+ 49757us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.967417 (+ 45895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4483283}
W20251023 20:15:39.969457 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 399) could send its response
I20251023 20:15:39.969487 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:39.969820 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 400) took 4670 ms. Trace:
I20251023 20:15:39.969952 25282 rpcz_store.cc:276] 1023 20:15:35.298984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.299031 (+ 47us) service_pool.cc:225] Handling call
1023 20:15:39.969611 (+4670580us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:39.969624 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.969776 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4670389}
W20251023 20:15:39.970274 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 400) could send its response
I20251023 20:15:39.970440 25282 tablet_replica.cc:333] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:39.969934 23790 tablet_bootstrap.cc:492] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:39.970892 25282 raft_consensus.cc:2243] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:39.976778 23787 tablet_bootstrap.cc:492] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:39.976923 23787 ts_tablet_manager.cc:1403] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20251023 20:15:39.977871 23666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e7909dba52748bdb78e08fa4b8a2d47" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.977959 25775 tablet_loader.cc:96] loaded metadata for tablet 385900c60d584c15b702f186ecfbc036 (table default.Table_49f2383c82e44457827030102d18d5bb [id=f49d21d543f5498580620fda6cfc97e9])
I20251023 20:15:39.978132 23787 raft_consensus.cc:359] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.978518 23787 raft_consensus.cc:385] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.978601 23787 raft_consensus.cc:740] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.978631 25726 raft_consensus.cc:493] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.978758 25726 raft_consensus.cc:515] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.978808 23787 consensus_queue.cc:260] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:39.979259 23787 ts_tablet_manager.cc:1434] T c7a63d37788240f0a0c3dbfdd14edeac P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251023 20:15:39.979262 25775 tablet_loader.cc:96] loaded metadata for tablet 38ce80a338d2494aac84fe6344d47e90 (table default.Table_5e48fffc45314e45a6f7ac8bba5ab31d [id=7260df2ce8034b73956131dd07502d7b])
I20251023 20:15:39.979312 25726 leader_election.cc:290] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.979511 23787 tablet_bootstrap.cc:492] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:39.971156 25282 raft_consensus.cc:2272] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:39.980590 25282 ts_tablet_manager.cc:1916] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:39.981806 23787 tablet_bootstrap.cc:654] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:39.983352 25775 tablet_loader.cc:96] loaded metadata for tablet 3a7306245d7b41cfab795450761144f5 (table default.Table_3ef388667712400bbd0f19475a8ed714 [id=adeeb85b87d747bd836be0b7c403ed13])
I20251023 20:15:39.984055 25775 tablet_loader.cc:96] loaded metadata for tablet 3a74b3f22b854c1ab2b6cc817b84cf47 (table default.Table_7eb5076c09144543885abb4392070ad8 [id=edda395d9ca84731a2bdc438a41aca1a])
I20251023 20:15:39.984905 25775 tablet_loader.cc:96] loaded metadata for tablet 3b25e1bd1f064b80a938be67961c1f63 (table default.Table_1e09690c826a4c028e998750992ec3d4 [id=f120045598ff41b09f32bbffde0e7092])
I20251023 20:15:39.984962 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a"
I20251023 20:15:39.985030 26020 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ae5a310b4a514c0f812bda133c756062 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251023 20:15:39.985921 25775 tablet_loader.cc:96] loaded metadata for tablet 3bb2bf696d2f4b639dfb0eff2a65e6f8 (table default.Table_10bf526d2893408887984427efbaa1f2 [id=933b756e69bf4d6d96590acd69f45267])
I20251023 20:15:39.989288 23786 tablet_bootstrap.cc:492] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:39.989423 23786 ts_tablet_manager.cc:1403] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.005s
I20251023 20:15:39.990056 23786 raft_consensus.cc:359] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.990201 23786 raft_consensus.cc:385] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:39.990240 23786 raft_consensus.cc:740] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:39.990573 23786 consensus_queue.cc:260] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.991176 26005 raft_consensus.cc:493] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:39.991608 23786 ts_tablet_manager.cc:1434] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251023 20:15:39.991918 26005 raft_consensus.cc:515] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.992856 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:39.993093 23668 raft_consensus.cc:2393] T dd07383242b74637bac46609f56c7e4d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:39.994181 23668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:39.994334 23668 raft_consensus.cc:2468] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:39.994696 23470 leader_election.cc:304] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:39.994931 25466 raft_consensus.cc:2804] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:39.994992 25466 raft_consensus.cc:493] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:39.995046 25466 raft_consensus.cc:3060] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:39.998071 25466 raft_consensus.cc:515] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:39.999372 25775 tablet_loader.cc:96] loaded metadata for tablet 3e7909dba52748bdb78e08fa4b8a2d47 (table default.Table_e2c91fae0bf148f198063a62b0426f63 [id=b4c7610c982345c4a6f90e2025b41b2d])
I20251023 20:15:39.999833 26005 leader_election.cc:290] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:39.999985 25466 leader_election.cc:290] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.000038 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:40.000198 23664 raft_consensus.cc:3060] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.004084 24780 ts_tablet_manager.cc:1929] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:40.004196 24780 log.cc:1199] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/c1f9935f5e4f4daa9df8fe1611ed76c9
I20251023 20:15:40.004459 23664 raft_consensus.cc:2468] T 63bdf5c80f2542f2a982d1de157d848d P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:40.004525 24780 ts_tablet_manager.cc:1950] T c1f9935f5e4f4daa9df8fe1611ed76c9 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:40.004949 23470 leader_election.cc:304] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.005298 25997 raft_consensus.cc:2804] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:40.005407 25997 raft_consensus.cc:697] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:40.005654 25997 consensus_queue.cc:237] T 63bdf5c80f2542f2a982d1de157d848d P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.008344 25466 consensus_queue.cc:1048] T 6ee1974eb5e343b7953dadda7d5799b1 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:40.009523 25776 raft_consensus.cc:493] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.009651 25776 raft_consensus.cc:515] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.010175 25776 leader_election.cc:290] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:40.010672 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0140a384382941a68e9a918d4a82a449" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.011073 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0140a384382941a68e9a918d4a82a449" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:40.011358 23786 tablet_bootstrap.cc:492] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
W20251023 20:15:40.011819 23605 leader_election.cc:343] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:40.011873 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 427) took 4660 ms. Trace:
I20251023 20:15:40.011937 25492 rpcz_store.cc:276] 1023 20:15:35.351680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.351717 (+ 37us) service_pool.cc:225] Handling call
1023 20:15:39.840489 (+4488772us) ts_tablet_manager.cc:1151] Deleting tablet f9bd2b109bfa4804a372965e60ba6e85
1023 20:15:39.840499 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.872146 (+ 31647us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:39.911178 (+ 39032us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.011854 (+100676us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4488640}
I20251023 20:15:40.012322 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 428) took 4660 ms. Trace:
I20251023 20:15:40.012390 25492 rpcz_store.cc:276] 1023 20:15:35.351963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.352002 (+ 39us) service_pool.cc:225] Handling call
1023 20:15:40.012210 (+4660208us) ts_tablet_manager.cc:1151] Deleting tablet 7f2a4f1d52984af58d6286671389b6a9
1023 20:15:40.012218 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.012307 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4660084}
I20251023 20:15:40.012677 25492 tablet_replica.cc:333] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:40.012858 25492 raft_consensus.cc:2243] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:40.012988 25492 raft_consensus.cc:2272] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.013377 23786 tablet_bootstrap.cc:654] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.013487 25492 ts_tablet_manager.cc:1916] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251023 20:15:40.017791 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 427) could send its response
W20251023 20:15:40.017906 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 428) could send its response
W20251023 20:15:40.018519 23604 leader_election.cc:343] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:40.018628 23604 leader_election.cc:304] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:40.019032 25776 raft_consensus.cc:2749] T 0140a384382941a68e9a918d4a82a449 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:40.021790 25492 ts_tablet_manager.cc:1929] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:40.021941 25492 log.cc:1199] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/e194d03a6aa946fbb2c79d91b196440b
I20251023 20:15:40.022285 25492 ts_tablet_manager.cc:1950] T e194d03a6aa946fbb2c79d91b196440b P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:40.023007 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 429) took 4670 ms. Trace:
I20251023 20:15:40.024422 25775 tablet_loader.cc:96] loaded metadata for tablet 44934f52cbeb448ba081b57cde783b2a (table default.Table_2dc2a5b423424735a1c370afde321275 [id=8f71d213abd24c078ff9db0f0b20c301])
I20251023 20:15:40.024824 25775 tablet_loader.cc:96] loaded metadata for tablet 44d6180be22943a0b753c0d204bc94eb (table default.Table_0c73476a1a6d41eeb3ed876c729c6d21 [id=619c9b0e1ec84349a682a425cbe85087])
I20251023 20:15:40.031601 23790 tablet_bootstrap.cc:654] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.031802 25492 rpcz_store.cc:276] 1023 20:15:35.352282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.352318 (+ 36us) service_pool.cc:225] Handling call
1023 20:15:40.012631 (+4660313us) ts_tablet_manager.cc:1151] Deleting tablet e194d03a6aa946fbb2c79d91b196440b
1023 20:15:40.012640 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.016614 (+ 3974us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.021692 (+ 5078us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.022989 (+ 1297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4660188}
W20251023 20:15:40.032603 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 429) could send its response
I20251023 20:15:40.033102 25492 tablet_replica.cc:333] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:40.035830 25492 raft_consensus.cc:2243] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:40.036338 26005 raft_consensus.cc:493] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.036496 26005 raft_consensus.cc:515] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.038038 23665 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "660d84eb63a4490886aba1777d449ee2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:40.038221 23665 raft_consensus.cc:2468] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:40.038619 23470 leader_election.cc:304] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.038938 25466 raft_consensus.cc:2804] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:40.039040 25466 raft_consensus.cc:493] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:40.039136 25466 raft_consensus.cc:3060] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.039376 23786 tablet_bootstrap.cc:492] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:40.039674 23786 ts_tablet_manager.cc:1403] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.001s
I20251023 20:15:40.040228 23787 tablet_bootstrap.cc:492] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:40.040356 23787 ts_tablet_manager.cc:1403] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.006s
I20251023 20:15:40.041051 23787 raft_consensus.cc:359] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.041241 23787 raft_consensus.cc:385] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.041302 23787 raft_consensus.cc:740] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.041508 23787 consensus_queue.cc:260] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.041469 23786 raft_consensus.cc:359] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.041833 23786 raft_consensus.cc:385] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.042009 23786 raft_consensus.cc:740] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.042096 25466 raft_consensus.cc:515] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.042511 23786 consensus_queue.cc:260] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.044140 23787 ts_tablet_manager.cc:1434] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251023 20:15:40.045019 23787 tablet_bootstrap.cc:492] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:40.045737 25775 tablet_loader.cc:96] loaded metadata for tablet 473324b22da14618a84aa1136d081993 (table default.Table_efbc2125b34c4ea48c56caa5c0b31909 [id=b08ba744dca54428bf1ef3851b865b4d])
I20251023 20:15:40.050493 25775 tablet_loader.cc:96] loaded metadata for tablet 4bce63866d524067b87145b07347ffcf (table default.Table_62d382e981754b839d2c87efc8004825 [id=25f85545812d46b9b41cfd5961a9a5a8])
I20251023 20:15:40.043099 26005 leader_election.cc:290] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.043951 23786 ts_tablet_manager.cc:1434] T ea0cbd6bec674cf5831486f500cc7b36 P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251023 20:15:40.051527 23786 tablet_bootstrap.cc:492] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:40.052198 25775 tablet_loader.cc:96] loaded metadata for tablet 4c0f207603f241669afd4523b0f9f82a (table default.Table_64ad686282f94cb7ac869f59933669f1 [id=91b7c90942ea4aa8bb8e4b32e6d7c5db])
I20251023 20:15:40.053005 25775 tablet_loader.cc:96] loaded metadata for tablet 4e72922aae98430da968874f44603317 (table default.Table_b4dc4f9658ce4837999abcf5720cea58 [id=ad988b9ca869431fb987604850e97fa8])
I20251023 20:15:40.054147 25776 raft_consensus.cc:493] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.054275 25776 raft_consensus.cc:515] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.055317 25775 tablet_loader.cc:96] loaded metadata for tablet 4f47ddde9f3d4042bc84853332e21950 (table default.Table_567481e14a7d4ef5a57941a4edc4694c [id=651489d40b6b425897b71f97e89c6dad])
I20251023 20:15:40.055193 25492 raft_consensus.cc:2272] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.055707 25776 leader_election.cc:290] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:40.056267 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2546fa8cb2004463af57f29ba28fdc1d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:40.056722 23604 leader_election.cc:343] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:40.056757 26005 raft_consensus.cc:493] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.056864 26005 raft_consensus.cc:515] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.057842 26005 leader_election.cc:290] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.057782 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2546fa8cb2004463af57f29ba28fdc1d" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
W20251023 20:15:40.060322 23605 leader_election.cc:343] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:40.060426 23605 leader_election.cc:304] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:40.060636 25776 raft_consensus.cc:2749] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:40.062014 23669 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3df1fd83e3c4ac29a7846736701044a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:40.062172 23669 raft_consensus.cc:2468] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:40.062682 23470 leader_election.cc:304] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.062970 26005 raft_consensus.cc:2804] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:40.063067 26005 raft_consensus.cc:493] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:40.063155 26005 raft_consensus.cc:3060] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.063831 25492 ts_tablet_manager.cc:1916] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:40.064714 23786 tablet_bootstrap.cc:654] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.065773 25775 tablet_loader.cc:96] loaded metadata for tablet 5161336598884742ad403fc88c9d9f0b (table default.Table_e1d99e08ee354f119f078584381958e6 [id=a34d678a3e1f41d8b65903bc2126e4e8])
I20251023 20:15:40.065745 26005 raft_consensus.cc:515] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.066903 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3df1fd83e3c4ac29a7846736701044a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:40.067104 23663 raft_consensus.cc:3060] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.067142 25282 ts_tablet_manager.cc:1929] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:40.067814 25282 log.cc:1199] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/ea3149f906064f8c8167292ec86540fd
I20251023 20:15:40.069149 26005 leader_election.cc:290] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.069962 25466 leader_election.cc:290] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.071983 23787 tablet_bootstrap.cc:654] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.072027 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "660d84eb63a4490886aba1777d449ee2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:40.072168 23664 raft_consensus.cc:3060] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.073618 25282 ts_tablet_manager.cc:1950] T ea3149f906064f8c8167292ec86540fd P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:40.074856 23664 raft_consensus.cc:2468] T 660d84eb63a4490886aba1777d449ee2 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:40.077469 23663 raft_consensus.cc:2468] T e3df1fd83e3c4ac29a7846736701044a P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:40.077956 23470 leader_election.cc:304] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.077991 25775 tablet_loader.cc:96] loaded metadata for tablet 53c04905c02a4f1b84fcb5426c597987 (table default.Table_e177ffb3adcc48dc84739ca2bdc1ae6e [id=44d2a2e9d5dc49be9b9474da1b1841bc])
I20251023 20:15:40.078243 23470 leader_election.cc:304] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.078495 26005 raft_consensus.cc:2804] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:40.078629 26005 raft_consensus.cc:697] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:40.078410 25775 tablet_loader.cc:96] loaded metadata for tablet 53f33020319d48c791e1156c67b55be6 (table default.Table_29c36300df674fe88a11b50c46a26f27 [id=3289908651fc4a63ba05b7b30c2ed718])
I20251023 20:15:40.078905 26005 consensus_queue.cc:237] T 660d84eb63a4490886aba1777d449ee2 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.079494 25775 tablet_loader.cc:96] loaded metadata for tablet 5407a850ec7045158f53cb8abf6fcb58 (table default.Table_61af7ade632c46a3becb1573b6313f0c [id=92ce6ecbc3fd4b48af6db9e587adcac5])
I20251023 20:15:40.082327 25997 raft_consensus.cc:2804] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:40.083138 25775 tablet_loader.cc:96] loaded metadata for tablet 559a70cf17ad4145acdb3033e01ad217 (table default.Table_12e6cb242b54467f9041afc60fa47eb3 [id=91a585d91b924838996b49fe349985f0])
I20251023 20:15:40.084017 25775 tablet_loader.cc:96] loaded metadata for tablet 58859629a2f14aacbdb5e2cd0edc27bc (table default.Table_f706cbf159424d419faa383ad30ebf77 [id=d399717713214f3dbc8839f2908ba34a])
I20251023 20:15:40.084868 25775 tablet_loader.cc:96] loaded metadata for tablet 59598ea40b9e4e5da0ec36f35f1094d6 (table default.Table_bbccd765c459437a838101139f1ec6c2 [id=9dd3c55753fa4ef5af5f31eddb6691b7])
I20251023 20:15:40.085839 25775 tablet_loader.cc:96] loaded metadata for tablet 59c9fe1fc8474c0d92d0964dfd5fd7a3 (table default.Table_24eda21d04d848c7b8971d30ac29f9f7 [id=30449f4a02de48c394825ffcd591455f])
I20251023 20:15:40.084519 25997 raft_consensus.cc:697] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:40.086423 25997 consensus_queue.cc:237] T e3df1fd83e3c4ac29a7846736701044a P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.087011 25775 tablet_loader.cc:96] loaded metadata for tablet 5a250341262142edb5b747a9e059ee54 (table default.Table_8587355637694585afe381e3e77c3958 [id=59862d7d13674aad83a3df0d07dce998])
I20251023 20:15:40.084740 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 401) took 4784 ms. Trace:
I20251023 20:15:40.088104 25776 raft_consensus.cc:493] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.088217 25776 raft_consensus.cc:515] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.088379 23787 tablet_bootstrap.cc:492] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:40.088488 23787 ts_tablet_manager.cc:1403] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.001s
I20251023 20:15:40.088809 25776 leader_election.cc:290] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:40.089251 23787 raft_consensus.cc:359] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.089330 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c0f207603f241669afd4523b0f9f82a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.089408 23787 raft_consensus.cc:385] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.089457 23787 raft_consensus.cc:740] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.089627 23787 consensus_queue.cc:260] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
W20251023 20:15:40.089995 23605 leader_election.cc:343] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:40.090137 25775 tablet_loader.cc:96] loaded metadata for tablet 5cdde6b251b14df4bf44142ce263b84a (table default.Table_30b62bd8bf6147a7a5800584576db269 [id=054cce203d7e4437a8a7aa22df487453])
I20251023 20:15:40.090186 23787 ts_tablet_manager.cc:1434] T 766882a030b446eb9406b3e32d903ee5 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:40.090606 25775 tablet_loader.cc:96] loaded metadata for tablet 5d801ca6ce63440abefe0c694889fa9e (table default.Table_cb399f5d5c1542dca90056ed300477cd [id=1a32884612a34df48b257e1a18bb5e73])
I20251023 20:15:40.090706 23787 tablet_bootstrap.cc:492] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:40.090875 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c0f207603f241669afd4523b0f9f82a" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
W20251023 20:15:40.091755 23604 leader_election.cc:343] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:40.091881 23604 leader_election.cc:304] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32; no voters: d220aba7c9c0400398ffde917376ee48, ea6853efbbcc484eb7529c7e9b37a47a
I20251023 20:15:40.092219 25776 raft_consensus.cc:2749] T 4c0f207603f241669afd4523b0f9f82a P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251023 20:15:40.092744 23787 tablet_bootstrap.cc:654] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.094439 25775 tablet_loader.cc:96] loaded metadata for tablet 5eab3e95be0f41dcb5a8ec63e91f5d8e (table default.Table_fc6b97c7acc94953be861d738e8f830a [id=84aabbb61e1c4d778b8e07ebdddbc477])
I20251023 20:15:40.089376 25282 rpcz_store.cc:276] 1023 20:15:35.299683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.299727 (+ 44us) service_pool.cc:225] Handling call
1023 20:15:39.970353 (+4670626us) ts_tablet_manager.cc:1151] Deleting tablet ea3149f906064f8c8167292ec86540fd
1023 20:15:39.970362 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.021688 (+ 51326us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.066369 (+ 44681us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.084490 (+ 18121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4670444}
W20251023 20:15:40.095073 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 401) could send its response
I20251023 20:15:40.096060 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 402) took 4795 ms. Trace:
I20251023 20:15:40.096673 25282 rpcz_store.cc:276] 1023 20:15:35.300810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.300915 (+ 105us) service_pool.cc:225] Handling call
1023 20:15:40.095445 (+4794530us) ts_tablet_manager.cc:1151] Deleting tablet 01fa5f9c700b458ea147373d78551573
1023 20:15:40.095621 (+ 176us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.096042 (+ 421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4794389}
W20251023 20:15:40.098210 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 402) could send its response
I20251023 20:15:40.098232 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 242) took 9045 ms. Trace:
I20251023 20:15:40.098399 24780 rpcz_store.cc:276] 1023 20:15:31.053023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.053159 (+ 136us) service_pool.cc:225] Handling call
1023 20:15:39.919535 (+8866376us) ts_tablet_manager.cc:1151] Deleting tablet c1f9935f5e4f4daa9df8fe1611ed76c9
1023 20:15:39.919544 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:39.924381 (+ 4837us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.003966 (+ 79585us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.098208 (+ 94242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8866225}
I20251023 20:15:40.099018 24780 tablet_replica.cc:333] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:40.098984 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 404) took 4781 ms. Trace:
I20251023 20:15:40.099368 24780 raft_consensus.cc:2243] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:40.099645 24780 raft_consensus.cc:2272] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.099313 25282 rpcz_store.cc:276] 1023 20:15:35.317297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.317346 (+ 49us) service_pool.cc:225] Handling call
1023 20:15:40.098197 (+4780851us) ts_tablet_manager.cc:1151] Deleting tablet 514cb3b1ed024041ac160a7d9acff618
1023 20:15:40.098304 (+ 107us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.098801 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4780679}
W20251023 20:15:40.100502 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 404) could send its response
I20251023 20:15:40.100785 24780 ts_tablet_manager.cc:1916] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:40.102921 25775 tablet_loader.cc:96] loaded metadata for tablet 6313bf6e463b42bd9990ec50904a3b60 (table default.Table_2860d1804bda4f67b048ee52ba46f15e [id=b9cb00bd8954475b909036fd9f123b1b])
I20251023 20:15:40.102319 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 405) took 4784 ms. Trace:
I20251023 20:15:40.104205 25775 tablet_loader.cc:96] loaded metadata for tablet 6313d3307aff4a6880176a06627e86fc (table default.Table_052bbf5fc61f4ddcb89f0c3efafb51c1 [id=3973b56caa79477f8542118421741084])
I20251023 20:15:40.104857 25775 tablet_loader.cc:96] loaded metadata for tablet 639450a251964b91b9291fdb029c4308 (table default.Table_05604aaaf6604c4c8f929ac257e5824e [id=04588122a9924ac5bcef60dedde8b1cd])
I20251023 20:15:40.106072 23786 tablet_bootstrap.cc:492] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a: No bootstrap required, opened a new log
I20251023 20:15:40.106004 25167 consensus_queue.cc:1048] T 58859629a2f14aacbdb5e2cd0edc27bc P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251023 20:15:40.106285 23786 ts_tablet_manager.cc:1403] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.001s
I20251023 20:15:40.106577 25775 tablet_loader.cc:96] loaded metadata for tablet 63bdf5c80f2542f2a982d1de157d848d (table default.Table_676121013e244d349484cd6c7cb75a8f [id=ddeeb413ab5b4a0ab8e01349eb17f066])
I20251023 20:15:40.107599 25775 tablet_loader.cc:96] loaded metadata for tablet 63ef1bae87c54aa988dfe49f828fe4f8 (table default.Table_6443f81020254f45a1b7fd93ad217e09 [id=4d43a0eda13041689a414ecbadb85e09])
I20251023 20:15:40.107705 23786 raft_consensus.cc:359] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.108033 23786 raft_consensus.cc:385] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.108130 23786 raft_consensus.cc:740] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.108512 23786 consensus_queue.cc:260] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.109503 25775 tablet_loader.cc:96] loaded metadata for tablet 6496f091681340f4971274646b8eaf7a (table default.Table_c2d9a9353b6944648cb319a6ea0cf2f6 [id=afe5584c068540c1915787d9b87cab33])
I20251023 20:15:40.109620 23786 ts_tablet_manager.cc:1434] T a0079496170849c6b3c233e1cd78e40e P ea6853efbbcc484eb7529c7e9b37a47a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251023 20:15:40.103008 25282 rpcz_store.cc:276] 1023 20:15:35.317393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.317550 (+ 157us) service_pool.cc:225] Handling call
1023 20:15:40.100834 (+4783284us) ts_tablet_manager.cc:1151] Deleting tablet 72408899f710470baaf6fdb268cb04c8
1023 20:15:40.100846 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.101991 (+ 1145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4782748}
W20251023 20:15:40.110301 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 405) could send its response
W20251023 20:15:40.110597 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 240) could send its response
W20251023 20:15:40.110730 23333 connection.cc:441] server connection from 127.0.0.1:38604 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 241) could send its response
I20251023 20:15:40.111203 23525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:40.112015 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 406) took 4794 ms. Trace:
I20251023 20:15:40.112874 25775 tablet_loader.cc:96] loaded metadata for tablet 6542aa8101914bf6a2f738ecd412012c (table default.Table_3448a0da3d5b4fffa69b2f473d4416b2 [id=492b5c03c69c46e3afcb57168577a59b])
I20251023 20:15:40.113543 25775 tablet_loader.cc:96] loaded metadata for tablet 660d84eb63a4490886aba1777d449ee2 (table default.Table_ec09021e59f8464f81cb6ceda1f1c567 [id=2fd818d424a44efdab6cf6fda7537ee8])
I20251023 20:15:40.112591 25282 rpcz_store.cc:276] 1023 20:15:35.317470 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.317754 (+ 284us) service_pool.cc:225] Handling call
1023 20:15:40.111195 (+4793441us) ts_tablet_manager.cc:1151] Deleting tablet c9bfbf0acdc84f2f8fe553694ee9897c
1023 20:15:40.111208 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.111759 (+ 551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4793085}
W20251023 20:15:40.114271 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 406) could send its response
I20251023 20:15:40.114686 23664 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7f9428277fb4ebf989fed7569dee687" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:40.115320 25282 tablet_replica.cc:333] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:40.117259 25775 tablet_loader.cc:96] loaded metadata for tablet 67e0701159c548c49e9e245820c45028 (table default.Table_7f5474847aa74113aca18fe96357be62 [id=ac08261334284759a3997df44867cc31])
I20251023 20:15:40.117728 25775 tablet_loader.cc:96] loaded metadata for tablet 68f5589f890f47e286a70419c2bf57c5 (table default.Table_5e2be9d05d7d431685748f79ce1c7311 [id=494f3980030c4d1fa22fec4ddcf14ff8])
I20251023 20:15:40.117395 25282 raft_consensus.cc:2243] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:40.118330 25775 tablet_loader.cc:96] loaded metadata for tablet 6a620b70bfcc4d29b326a4c63e16b3c5 (table default.Table_8d1521c5f05d4bb5a060ed68c5fa6ca5 [id=2f0aa245600741b3a778c959b60b936b])
I20251023 20:15:40.118201 25282 raft_consensus.cc:2272] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.118971 25775 tablet_loader.cc:96] loaded metadata for tablet 6abfda15e6c242b588d4ed4401008156 (table default.Table_15836e4048de43f79d9a1d06e0072c3c [id=b9d64b8c30014c5abbb2312d1b8017fe])
I20251023 20:15:40.122032 25775 tablet_loader.cc:96] loaded metadata for tablet 6e4189623c7a4ca1bdf73f71663e2b77 (table default.Table_49be7be29ce24a9ca4f5cc29644bc6c2 [id=d236ba1e610441a8886230b0365ee300])
I20251023 20:15:40.122432 25775 tablet_loader.cc:96] loaded metadata for tablet 6ecfa16e75f543fab12fa0bac4d9e3e8 (table default.Table_34d8a4d24be94b1ea676b7a0df81b96c [id=7b824b39feed48148896732770542d20])
I20251023 20:15:40.122856 25775 tablet_loader.cc:96] loaded metadata for tablet 6ee1974eb5e343b7953dadda7d5799b1 (table default.Table_515e6170d0e34340a74759ab0a056f72 [id=95f5faaee2b947fea16c52c55657db4a])
I20251023 20:15:40.123795 25282 ts_tablet_manager.cc:1916] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:40.128171 25466 raft_consensus.cc:493] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.128340 25466 raft_consensus.cc:515] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.129505 25466 leader_election.cc:290] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
W20251023 20:15:40.129927 23470 leader_election.cc:343] T 453eea5cdd2e47f29577c61b40ee59c0 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211): Not found: Tablet not found: 453eea5cdd2e47f29577c61b40ee59c0
I20251023 20:15:40.130170 25775 tablet_loader.cc:96] loaded metadata for tablet 72d1dde8b0ac4812869ef233c18e550f (table default.Table_efda7d26e2124ec4b1eeca32fe1b545e [id=6f9ad1c9f64a44a4b856013bb8fd4bd1])
I20251023 20:15:40.131356 25775 tablet_loader.cc:96] loaded metadata for tablet 744d31fccbbe4fa59c56e24136e3b2b9 (table default.Table_2bc741a605f74c0dbcc7aecc45137db4 [id=ec622f1a57064ff2a6ba1abb48b1181e])
I20251023 20:15:40.141011 23786 tablet_bootstrap.cc:492] T 22b7598fe6734d9098fcaa0a0f4bdc7b P ea6853efbbcc484eb7529c7e9b37a47a: Bootstrap starting.
I20251023 20:15:40.145335 23786 tablet_bootstrap.cc:654] T 22b7598fe6734d9098fcaa0a0f4bdc7b P ea6853efbbcc484eb7529c7e9b37a47a: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.147955 25775 tablet_loader.cc:96] loaded metadata for tablet 766882a030b446eb9406b3e32d903ee5 (table default.Table_511899f307ee49fb8612942d4b966ee2 [id=12648eb0bb8b4909814e3f2cb9cec1ae])
I20251023 20:15:40.151667 23790 tablet_bootstrap.cc:492] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:40.152890 23787 tablet_bootstrap.cc:492] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:40.153481 25775 tablet_loader.cc:96] loaded metadata for tablet 82ae1a94b011412c851cd96946744fc4 (table default.Table_c4502b0740ee4a34a1091754a3c9adf3 [id=c830d9df19924cb09bb38ff2d445b3fc])
I20251023 20:15:40.154364 23790 ts_tablet_manager.cc:1403] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.187s user 0.001s sys 0.005s
I20251023 20:15:40.154654 23787 ts_tablet_manager.cc:1403] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20251023 20:15:40.156172 23787 raft_consensus.cc:359] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.156240 23790 raft_consensus.cc:359] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.156711 23790 raft_consensus.cc:385] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.156845 23790 raft_consensus.cc:740] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.159339 23396 raft_consensus.cc:3060] T 2c40164e687545c5b2d2751baaee4571 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.159477 23787 raft_consensus.cc:385] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.162606 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd07383242b74637bac46609f56c7e4d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.162799 23396 raft_consensus.cc:2393] T dd07383242b74637bac46609f56c7e4d P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea6853efbbcc484eb7529c7e9b37a47a in current term 1: Already voted for candidate d220aba7c9c0400398ffde917376ee48 in this term.
I20251023 20:15:40.162784 23787 raft_consensus.cc:740] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.163215 25775 tablet_loader.cc:96] loaded metadata for tablet 866f7e4cbed341b7b75182891c034c28 (table default.Table_6ea21ba3f1684634907e258298159baf [id=5962bfcedadc486d8f5e985d1c213fe2])
I20251023 20:15:40.163347 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b30e8ca2c96f4063a7f0e8f324c78b12" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.163560 23396 raft_consensus.cc:3060] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.163573 23787 consensus_queue.cc:260] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.164389 25775 tablet_loader.cc:96] loaded metadata for tablet 878508d6bf0d4a59b6f64155361b4418 (table default.Table_3abcf10252074989b84e5566c762a99d [id=ba74b6d39572458e9f01923f709c2d38])
I20251023 20:15:40.165575 23787 ts_tablet_manager.cc:1434] T 139c1bb5ffa743d4a62edf5e98932454 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251023 20:15:40.165788 25921 raft_consensus.cc:493] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.165933 25921 raft_consensus.cc:515] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.166529 25921 leader_election.cc:290] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.166736 23396 raft_consensus.cc:2468] T b30e8ca2c96f4063a7f0e8f324c78b12 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:40.166901 23787 tablet_bootstrap.cc:492] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:40.167122 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "255521d943d34713aaf773b15f9168c6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:40.167280 23663 raft_consensus.cc:2468] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:40.167644 25775 tablet_loader.cc:96] loaded metadata for tablet 8827ec39f42a423bb5052b5fd4b27334 (table default.Table_0984ad1922884cb19d48dc21a903dfe2 [id=8b419bc021804ce7ab372b0515b672f2])
I20251023 20:15:40.167665 23470 leader_election.cc:304] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.167912 25921 raft_consensus.cc:2804] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:40.167986 25921 raft_consensus.cc:493] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:40.168052 25921 raft_consensus.cc:3060] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.168252 25775 tablet_loader.cc:96] loaded metadata for tablet 882f7bf08e2f4b5382702c3d6eaf201e (table default.Table_39fdd0d19cae4e928fdee07e74900b4b [id=356410224c0f44bdacbaa543e9dee27d])
I20251023 20:15:40.169516 23787 tablet_bootstrap.cc:654] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.169852 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88f1df552f93427e91ca19c8170d69bd" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.170055 25775 tablet_loader.cc:96] loaded metadata for tablet 886838284a0548f6bb089f62c686165e (table default.Table_95fb2c2528f44d8d84f4cd67d9663502 [id=4867e660fb23438495400e364dba7b48])
I20251023 20:15:40.170485 25775 tablet_loader.cc:96] loaded metadata for tablet 88f1df552f93427e91ca19c8170d69bd (table default.Table_39425bc1be794ee8b1283c048094c708 [id=b9216639dc7c4f0380fb618444206fae])
I20251023 20:15:40.170913 25775 tablet_loader.cc:96] loaded metadata for tablet 8a45d9b3031b46d390e0a1010489c718 (table default.Table_ea656dd8eee44e5f89ad208c70cddc39 [id=96188ab2961f4b14b71df728d91aa4e3])
I20251023 20:15:40.170887 25921 raft_consensus.cc:515] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.171986 25921 leader_election.cc:290] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.172245 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "255521d943d34713aaf773b15f9168c6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:40.172390 23663 raft_consensus.cc:3060] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.172444 25775 tablet_loader.cc:96] loaded metadata for tablet 8a630b83f614447ebf2a08851d6b827d (table default.Table_62817ee2bac14a6a8c254e60d7836061 [id=568404f36000475ab792fa4321f99249])
I20251023 20:15:40.172783 25775 tablet_loader.cc:96] loaded metadata for tablet 8d22aa40e5844d78a8679a3ad1b08da9 (table default.Table_b342acb3e1f447b8985fa86b3f8c7fc4 [id=79163b61e42a43d8879399cd5f31ce44])
I20251023 20:15:40.173136 25775 tablet_loader.cc:96] loaded metadata for tablet 8d9b192c111c47d1983e9a48d07da288 (table default.Table_d49e3a9e4cfd4d0b90efbbae5c25537d [id=5a207d264756433f94af5164462692ff])
I20251023 20:15:40.173913 25775 tablet_loader.cc:96] loaded metadata for tablet 8e541aeb3cb54d19a673b541a8a9c127 (table default.Table_d55345cd0dd44edaa55e9e412037691f [id=4da4a7df78da46b18a39004985932e04])
I20251023 20:15:40.174237 25775 tablet_loader.cc:96] loaded metadata for tablet 8e718cb7205643fd8c3045c3fab30d3e (table default.Table_3eeb50ae897f4006ad63033f2ad0324a [id=8477c10fb2f4422996d4eab8717d7fe8])
I20251023 20:15:40.174721 23663 raft_consensus.cc:2468] T 255521d943d34713aaf773b15f9168c6 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:40.175207 23470 leader_election.cc:304] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.175454 25921 raft_consensus.cc:2804] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:40.177896 25921 raft_consensus.cc:697] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:40.178261 25921 consensus_queue.cc:237] T 255521d943d34713aaf773b15f9168c6 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.178603 25775 tablet_loader.cc:96] loaded metadata for tablet 9047d641796b4f20b0de82b277ce877c (table default.Table_e850560a68bd47a1bfb2bff80151368a [id=ded4286edf6742b385a5b3d01c4f4525])
I20251023 20:15:40.179463 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.179654 23391 raft_consensus.cc:2468] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:40.180076 25775 tablet_loader.cc:96] loaded metadata for tablet 90d58bceecba442898a3d742fa49a4a2 (table default.Table_e446e75e73024a1fb40206c4a4c835ac [id=e73714e917184f788bf57815006ec75c])
I20251023 20:15:40.180672 25775 tablet_loader.cc:96] loaded metadata for tablet 92576756ec574ceaa4c81f435cb72301 (table default.Table_eb73fed1f9bb425daff2e3278ad80ece [id=fba1e090c19d406c8c9c4728885f1e28])
I20251023 20:15:40.183375 25489 raft_consensus.cc:493] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.183512 25489 raft_consensus.cc:515] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.184082 25489 leader_election.cc:290] T 95a06da281194f95b126483314f6debd P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.192727 25775 tablet_loader.cc:96] loaded metadata for tablet 94631fca1b834bc89791cb07fdc93523 (table default.Table_43421f044a504b98a442f09d4fc5c601 [id=7bab3a0ea7c2419e88d292294d15ba48])
I20251023 20:15:40.193809 25775 tablet_loader.cc:96] loaded metadata for tablet 94b4838644a9403693572350b1bdd0e2 (table default.Table_222ab66f7150462aad2efe8d16484b94 [id=f52ec73cfcfe4b36a08422d69aabf321])
I20251023 20:15:40.194741 23787 tablet_bootstrap.cc:492] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:40.194954 23787 ts_tablet_manager.cc:1403] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20251023 20:15:40.195672 23787 raft_consensus.cc:359] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.195904 23787 raft_consensus.cc:385] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.195988 23787 raft_consensus.cc:740] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.196241 23787 consensus_queue.cc:260] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.197180 23787 ts_tablet_manager.cc:1434] T 639450a251964b91b9291fdb029c4308 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251023 20:15:40.197887 23787 tablet_bootstrap.cc:492] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:40.200009 23787 tablet_bootstrap.cc:654] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.200708 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb2bf696d2f4b639dfb0eff2a65e6f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.201368 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.201514 23391 raft_consensus.cc:3060] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.203896 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bb2bf696d2f4b639dfb0eff2a65e6f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.204064 23391 raft_consensus.cc:2468] T aec7c5ea622949aca6c3dbe0e5f8e81e P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:40.206360 23391 raft_consensus.cc:3060] T 3e7909dba52748bdb78e08fa4b8a2d47 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.207767 25492 ts_tablet_manager.cc:1929] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:40.207938 25492 log.cc:1199] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/c1f9935f5e4f4daa9df8fe1611ed76c9
I20251023 20:15:40.208969 25775 tablet_loader.cc:96] loaded metadata for tablet 94bb861a96cd40748d35e909418b813a (table default.Table_d0548a52907a4b1984665988fa00b425 [id=de89b68b2c194fada173999f7a0af035])
I20251023 20:15:40.209474 25775 tablet_loader.cc:96] loaded metadata for tablet 94cd8880a78c4b4eb0c19ba5944b12f2 (table default.Table_240b16ca6e1543f2aab607c1d6a447fd [id=4aa2d573bbb34a3ebda2ac6376a387ea])
I20251023 20:15:40.210994 25775 tablet_loader.cc:96] loaded metadata for tablet 955ec4740c4b4d3fa19a20faeaebc09e (table default.Table_aeba123341a14c5aba5a1ccca37e5395 [id=5b8f061cbf014eed80a09b13ee85429f])
I20251023 20:15:40.211452 25775 tablet_loader.cc:96] loaded metadata for tablet 95a06da281194f95b126483314f6debd (table default.Table_69f62f4882e04aa1a32092b258c6acaa [id=ee35b2f67f5f4d9488b1c6484ca4343a])
I20251023 20:15:40.211872 25775 tablet_loader.cc:96] loaded metadata for tablet 9749c6f3fe5545ab93bf49aae97e9818 (table default.Table_b589e93e3314462d8ef529985314e501 [id=bdeb476ba31d425893262ab98d5b579f])
I20251023 20:15:40.212316 25775 tablet_loader.cc:96] loaded metadata for tablet 975fc4848c8f4db4bfd2dad9ac620d83 (table default.Table_0e4d6c666b2d4372afd985345a92edd4 [id=98e1db67295f441c9904194cba2da98b])
I20251023 20:15:40.213011 25775 tablet_loader.cc:96] loaded metadata for tablet 98a4553c1f014ef3ac975bf7a8076334 (table default.Table_ac4b704784f2423a82ad591bbed856ab [id=3cf9ba4080ce4e5ead110ed71e43ce1f])
I20251023 20:15:40.157240 23790 consensus_queue.cc:260] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.219980 25282 ts_tablet_manager.cc:1929] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:40.221330 25282 log.cc:1199] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-2/wal/wals/ea0cbd6bec674cf5831486f500cc7b36
I20251023 20:15:40.222324 25492 ts_tablet_manager.cc:1950] T c1f9935f5e4f4daa9df8fe1611ed76c9 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:40.224521 23790 ts_tablet_manager.cc:1434] T b3a216aeebe649f79b6d873e1c59df37 P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.070s user 0.000s sys 0.002s
I20251023 20:15:40.225836 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.228996 25282 ts_tablet_manager.cc:1950] T ea0cbd6bec674cf5831486f500cc7b36 P 9bc6522db11e4960a9aad84435371a32: Deleting consensus metadata
I20251023 20:15:40.229954 23790 tablet_bootstrap.cc:492] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:40.234088 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63bdf5c80f2542f2a982d1de157d848d" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.234838 23790 tablet_bootstrap.cc:654] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.235437 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "660d84eb63a4490886aba1777d449ee2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.236474 23787 tablet_bootstrap.cc:492] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:40.236670 23787 ts_tablet_manager.cc:1403] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.002s
I20251023 20:15:40.237469 23787 raft_consensus.cc:359] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.237689 23787 raft_consensus.cc:385] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.237782 23787 raft_consensus.cc:740] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.238076 23787 consensus_queue.cc:260] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.238891 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 408) took 4920 ms. Trace:
I20251023 20:15:40.240214 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3df1fd83e3c4ac29a7846736701044a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.240978 24780 ts_tablet_manager.cc:1929] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:40.241155 24780 log.cc:1199] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/857da3642f0846ce92aaed0491e89644
I20251023 20:15:40.241535 24780 ts_tablet_manager.cc:1950] T 857da3642f0846ce92aaed0491e89644 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:40.240123 25282 rpcz_store.cc:276] 1023 20:15:35.317655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.317898 (+ 243us) service_pool.cc:225] Handling call
1023 20:15:40.115028 (+4797130us) ts_tablet_manager.cc:1151] Deleting tablet ea0cbd6bec674cf5831486f500cc7b36
1023 20:15:40.115043 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.170370 (+ 55327us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.218432 (+ 48062us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.238570 (+ 20138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4796772}
W20251023 20:15:40.242295 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 408) could send its response
I20251023 20:15:40.242751 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 243) took 9189 ms. Trace:
I20251023 20:15:40.238984 23787 ts_tablet_manager.cc:1434] T 2d8ded8261ed406d88df394420730a31 P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251023 20:15:40.243597 23787 tablet_bootstrap.cc:492] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:40.243860 25282 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 407) took 4926 ms. Trace:
I20251023 20:15:40.246840 25775 tablet_loader.cc:96] loaded metadata for tablet 9ac11204192a4b40963c0d241221d1cb (table default.Table_572a1f2b4ae0442ba59eb601fffb021c [id=753d0b7ee3444bb9bfe444b63a1ac511])
I20251023 20:15:40.247702 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 430) took 4895 ms. Trace:
I20251023 20:15:40.250825 25282 rpcz_store.cc:276] 1023 20:15:35.317546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.318041 (+ 495us) service_pool.cc:225] Handling call
1023 20:15:40.243033 (+4924992us) ts_tablet_manager.cc:1151] Deleting tablet 05933cb9433d48d285b2db82b289bdc8
1023 20:15:40.243045 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.243838 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4924644}
W20251023 20:15:40.253260 23603 connection.cc:441] server connection from 127.0.0.1:60202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60202 (request call id 407) could send its response
I20251023 20:15:40.254555 25282 tablet_replica.cc:333] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32: stopping tablet replica
I20251023 20:15:40.257165 25282 raft_consensus.cc:2243] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:40.257705 25492 rpcz_store.cc:276] 1023 20:15:35.352603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.352705 (+ 102us) service_pool.cc:225] Handling call
1023 20:15:40.033033 (+4680328us) ts_tablet_manager.cc:1151] Deleting tablet c1f9935f5e4f4daa9df8fe1611ed76c9
1023 20:15:40.033045 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.204702 (+171657us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.207682 (+ 2980us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.247679 (+ 39997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4680198}
W20251023 20:15:40.258098 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 430) could send its response
I20251023 20:15:40.258436 25492 tablet_replica.cc:333] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:40.258661 25492 raft_consensus.cc:2243] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:40.258877 25492 raft_consensus.cc:2272] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.258692 25282 raft_consensus.cc:2272] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.259549 25492 ts_tablet_manager.cc:1916] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:40.264521 25282 ts_tablet_manager.cc:1916] T 2546fa8cb2004463af57f29ba28fdc1d P 9bc6522db11e4960a9aad84435371a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:40.245240 23787 tablet_bootstrap.cc:654] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.269976 25775 tablet_loader.cc:96] loaded metadata for tablet 9b766ff2dff34a8f80f70591dba54017 (table default.Table_4e035ee3714e4e47b7748fa23656760a [id=32d1b2b4b0c84ec48a98220a4def4664])
I20251023 20:15:40.270517 25775 tablet_loader.cc:96] loaded metadata for tablet 9c0aeac2f20c4ca48c130a0c851f290f (table default.Table_3f90e08382b448498312f1fade68ae16 [id=a7821323b5954830ad19f8f771bfe6c6])
I20251023 20:15:40.271196 25775 tablet_loader.cc:96] loaded metadata for tablet 9e76f2e26dcd4cdfbf72de899f5a1458 (table default.Table_d1b58c19a42b4416a91f2614ebc3ea6b [id=b0c6ab2358b84be2862d31f863e3c32d])
I20251023 20:15:40.271613 25775 tablet_loader.cc:96] loaded metadata for tablet 9ea980533a5241f5bde89c7cc4c72f8d (table default.Table_9eb13d4c90b44054a9f549bd289a9c6e [id=f97ed62dc77d4f8991056b5268dd88df])
I20251023 20:15:40.272567 25775 tablet_loader.cc:96] loaded metadata for tablet a0079496170849c6b3c233e1cd78e40e (table default.Table_3a5e9b90e86340ed80fdb4a8679777e6 [id=22d88af65ebd43389f6e58573143816c])
I20251023 20:15:40.273001 25775 tablet_loader.cc:96] loaded metadata for tablet a04c65155ce048cb82b150cbe20be849 (table default.Table_acb6fd552fa646929598fb1d07e2bb12 [id=e58f31a7c86c478f92c6471e1020585b])
I20251023 20:15:40.245240 23391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3df1fd83e3c4ac29a7846736701044a" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.257454 23396 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "660d84eb63a4490886aba1777d449ee2" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.274391 24780 rpcz_store.cc:276] 1023 20:15:31.053125 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.053341 (+ 216us) service_pool.cc:225] Handling call
1023 20:15:40.098897 (+9045556us) ts_tablet_manager.cc:1151] Deleting tablet 857da3642f0846ce92aaed0491e89644
1023 20:15:40.098908 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.203251 (+104343us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.240867 (+ 37616us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.242734 (+ 1867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9045426}
I20251023 20:15:40.268489 23398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453eea5cdd2e47f29577c61b40ee59c0" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.274770 24780 tablet_replica.cc:333] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:40.269443 23790 tablet_bootstrap.cc:492] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48: No bootstrap required, opened a new log
I20251023 20:15:40.271612 23393 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "255521d943d34713aaf773b15f9168c6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48" is_pre_election: true
I20251023 20:15:40.273723 23388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "255521d943d34713aaf773b15f9168c6" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"
I20251023 20:15:40.275090 24780 raft_consensus.cc:2243] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:40.275085 23790 ts_tablet_manager.cc:1403] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.002s
I20251023 20:15:40.275430 23391 raft_consensus.cc:3060] T a7f9428277fb4ebf989fed7569dee687 P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.276077 23790 raft_consensus.cc:359] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.276964 23787 tablet_bootstrap.cc:492] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32: No bootstrap required, opened a new log
I20251023 20:15:40.277158 23787 ts_tablet_manager.cc:1403] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.002s
I20251023 20:15:40.277973 23787 raft_consensus.cc:359] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.278185 23790 raft_consensus.cc:385] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.278278 23790 raft_consensus.cc:740] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d220aba7c9c0400398ffde917376ee48, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.278579 23790 consensus_queue.cc:260] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.279495 23790 ts_tablet_manager.cc:1434] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251023 20:15:40.279618 23787 raft_consensus.cc:385] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251023 20:15:40.279727 23787 raft_consensus.cc:740] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc6522db11e4960a9aad84435371a32, State: Initialized, Role: FOLLOWER
I20251023 20:15:40.279983 23787 consensus_queue.cc:260] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.280472 23790 tablet_bootstrap.cc:492] T 22b7598fe6734d9098fcaa0a0f4bdc7b P d220aba7c9c0400398ffde917376ee48: Bootstrap starting.
I20251023 20:15:40.283561 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:40.284176 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aec7c5ea622949aca6c3dbe0e5f8e81e" candidate_uuid: "d220aba7c9c0400398ffde917376ee48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:40.284358 23663 raft_consensus.cc:2393] T aec7c5ea622949aca6c3dbe0e5f8e81e P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d220aba7c9c0400398ffde917376ee48 in current term 1: Already voted for candidate ea6853efbbcc484eb7529c7e9b37a47a in this term.
I20251023 20:15:40.287196 23787 ts_tablet_manager.cc:1434] T abdad7d76eb8441496d38642382f74bb P 9bc6522db11e4960a9aad84435371a32: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251023 20:15:40.288724 25921 raft_consensus.cc:493] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.288857 25921 raft_consensus.cc:515] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.289546 25921 leader_election.cc:290] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.290508 25492 ts_tablet_manager.cc:1929] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251023 20:15:40.290635 25492 log.cc:1199] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-1/wal/wals/7e36bb5e61d34b35a373bd6a51a7ac5e
I20251023 20:15:40.290997 25492 ts_tablet_manager.cc:1950] T 7e36bb5e61d34b35a373bd6a51a7ac5e P ea6853efbbcc484eb7529c7e9b37a47a: Deleting consensus metadata
I20251023 20:15:40.295274 23790 tablet_bootstrap.cc:654] T 22b7598fe6734d9098fcaa0a0f4bdc7b P d220aba7c9c0400398ffde917376ee48: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.291689 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63ef1bae87c54aa988dfe49f828fe4f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32" is_pre_election: true
I20251023 20:15:40.295483 23663 raft_consensus.cc:2468] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 0.
I20251023 20:15:40.295948 23470 leader_election.cc:304] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.296245 25925 raft_consensus.cc:2804] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251023 20:15:40.296315 25925 raft_consensus.cc:493] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251023 20:15:40.296372 25925 raft_consensus.cc:3060] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.297215 25492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 431) took 4944 ms. Trace:
I20251023 20:15:40.297324 25492 rpcz_store.cc:276] 1023 20:15:35.352973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:35.353131 (+ 158us) service_pool.cc:225] Handling call
1023 20:15:40.258386 (+4905255us) ts_tablet_manager.cc:1151] Deleting tablet 7e36bb5e61d34b35a373bd6a51a7ac5e
1023 20:15:40.258398 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.281797 (+ 23399us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.290407 (+ 8610us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.297196 (+ 6789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4905123}
W20251023 20:15:40.297590 23469 connection.cc:441] server connection from 127.0.0.1:48312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48312 (request call id 431) could send its response
I20251023 20:15:40.297660 25492 tablet_replica.cc:333] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a: stopping tablet replica
I20251023 20:15:40.280196 24780 raft_consensus.cc:2272] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.297848 25492 raft_consensus.cc:2243] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Raft consensus shutting down.
I20251023 20:15:40.298202 25492 raft_consensus.cc:2272] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251023 20:15:40.298529 24780 ts_tablet_manager.cc:1916] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:40.298820 25925 raft_consensus.cc:515] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.298887 24821 raft_consensus.cc:493] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.299002 24821 raft_consensus.cc:515] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.299330 25925 leader_election.cc:290] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Requested vote from peers d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.299569 24821 leader_election.cc:290] T 118be7aa6380469db091d2788472e62b P d220aba7c9c0400398ffde917376ee48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), 9bc6522db11e4960a9aad84435371a32 (127.22.127.131:46211)
I20251023 20:15:40.300105 23663 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63ef1bae87c54aa988dfe49f828fe4f8" candidate_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc6522db11e4960a9aad84435371a32"
I20251023 20:15:40.300246 23663 raft_consensus.cc:3060] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Advancing to term 1
I20251023 20:15:40.302634 23663 raft_consensus.cc:2468] T 63ef1bae87c54aa988dfe49f828fe4f8 P 9bc6522db11e4960a9aad84435371a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6853efbbcc484eb7529c7e9b37a47a in term 1.
I20251023 20:15:40.303081 23470 leader_election.cc:304] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc6522db11e4960a9aad84435371a32, ea6853efbbcc484eb7529c7e9b37a47a; no voters:
I20251023 20:15:40.303320 25997 raft_consensus.cc:2804] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 FOLLOWER]: Leader election won for term 1
I20251023 20:15:40.303421 25997 raft_consensus.cc:697] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [term 1 LEADER]: Becoming Leader. State: Replica: ea6853efbbcc484eb7529c7e9b37a47a, State: Running, Role: LEADER
I20251023 20:15:40.303646 25997 consensus_queue.cc:237] T 63ef1bae87c54aa988dfe49f828fe4f8 P ea6853efbbcc484eb7529c7e9b37a47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } }
I20251023 20:15:40.305466 25775 tablet_loader.cc:96] loaded metadata for tablet a2c022d78cac4959984d608d64df851b (table default.Table_698f8853c4774ccdae2e37d28f90f368 [id=3e597b14c5d944838c32914dc1779d20])
I20251023 20:15:40.308647 24780 ts_tablet_manager.cc:1929] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251023 20:15:40.308772 24780 log.cc:1199] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: Deleting WAL directory at /tmp/dist-test-taskvdN7pO/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761250510763983-23038-0/minicluster-data/ts-0/wal/wals/a841ffdc6cdd432dac7afe2e2b53f224
I20251023 20:15:40.309046 24780 ts_tablet_manager.cc:1950] T a841ffdc6cdd432dac7afe2e2b53f224 P d220aba7c9c0400398ffde917376ee48: Deleting consensus metadata
I20251023 20:15:40.313735 24780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38604 (request call id 244) took 9259 ms. Trace:
I20251023 20:15:40.314497 23787 tablet_bootstrap.cc:492] T 3b25e1bd1f064b80a938be67961c1f63 P 9bc6522db11e4960a9aad84435371a32: Bootstrap starting.
I20251023 20:15:40.313838 24780 rpcz_store.cc:276] 1023 20:15:31.053953 (+ 0us) service_pool.cc:168] Inserting onto call queue
1023 20:15:31.053995 (+ 42us) service_pool.cc:225] Handling call
1023 20:15:40.274723 (+9220728us) ts_tablet_manager.cc:1151] Deleting tablet a841ffdc6cdd432dac7afe2e2b53f224
1023 20:15:40.274733 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1023 20:15:40.301836 (+ 27103us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.308535 (+ 6699us) tablet_metadata.cc:641] Metadata flushed
1023 20:15:40.313712 (+ 5177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":755,"tablet-delete.queue_time_us":9220534}
I20251023 20:15:40.316025 24780 tablet_replica.cc:333] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48: stopping tablet replica
I20251023 20:15:40.316203 24780 raft_consensus.cc:2243] T 39a62c97e0a3459e91c24c1c2f92389a P d220aba7c9c0400398ffde917376ee48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251023 20:15:40.316396 25776 raft_consensus.cc:493] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251023 20:15:40.316448 25492 ts_tablet_manager.cc:1916] T 9a6a2d2b25824dbdafaf2a9cabb0f061 P ea6853efbbcc484eb7529c7e9b37a47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251023 20:15:40.316526 25776 raft_consensus.cc:515] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" member_type: VOTER last_known_addr { host: "127.22.127.130" port: 34173 } } peers { permanent_uuid: "9bc6522db11e4960a9aad84435371a32" member_type: VOTER last_known_addr { host: "127.22.127.131" port: 46211 } } peers { permanent_uuid: "d220aba7c9c0400398ffde917376ee48" member_type: VOTER last_known_addr { host: "127.22.127.129" port: 34961 } }
I20251023 20:15:40.317098 25776 leader_election.cc:290] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173), d220aba7c9c0400398ffde917376ee48 (127.22.127.129:34961)
I20251023 20:15:40.317226 23787 tablet_bootstrap.cc:654] T 3b25e1bd1f064b80a938be67961c1f63 P 9bc6522db11e4960a9aad84435371a32: Neither blocks nor log segments found. Creating new log.
I20251023 20:15:40.309872 25775 tablet_loader.cc:96] loaded metadata for tablet a2d4ba34cc444baf9fe725bc81478517 (table default.Table_87caa0c69120413a92c6b65337baad14 [id=3227f5fd7d78482c99fb9f654ba8e3c6])
I20251023 20:15:40.317741 23534 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a45d01f99d2048d18bdaca7465519030" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6853efbbcc484eb7529c7e9b37a47a" is_pre_election: true
I20251023 20:15:40.318317 25775 tablet_loader.cc:96] loaded metadata for tablet a338b06b56a9482b81944f4fce9580f9 (table default.Table_331b508de7624556842102252e4ccc38 [id=4a9c4e7500ab40f5b0daf197ce160eb5])
W20251023 20:15:40.318586 23604 leader_election.cc:343] T a45d01f99d2048d18bdaca7465519030 P 9bc6522db11e4960a9aad84435371a32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea6853efbbcc484eb7529c7e9b37a47a (127.22.127.130:34173): Illegal state: must be running to vote when last-logged opid is not known
I20251023 20:15:40.319145 23388 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a45d01f99d2048d18bdaca7465519030" candidate_uuid: "9bc6522db11e4960a9aad84435371a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d220aba7c9c0400398ffde917376ee48"